Oct 30 12:31:49.301626 ip-10-0-134-193 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:31:49.694844 ip-10-0-134-193 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:31:49.694844 ip-10-0-134-193 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:31:49.694844 ip-10-0-134-193 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:31:49.694844 ip-10-0-134-193 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:31:49.700191 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700041 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:31:49.700191 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700186 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:31:49.700191 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700192 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:31:49.700191 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700197 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700202 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700213 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700217 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700222 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700226 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700230 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700234 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700238 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700243 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700251 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700258 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700263 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700281 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700292 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700297 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700301 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700307 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700320 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:31:49.700450 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700362 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700369 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700374 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700379 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700384 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700389 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700394 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700399 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700404 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700408 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700418 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700430 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700435 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700439 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700444 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700449 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700453 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700457 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:31:49.700918 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700471 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.700997 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701002 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701005 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701009 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701012 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701015 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701018 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701021 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701024 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701027 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701029 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701033 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701036 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701038 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701041 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701044 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701051 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701054 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701056 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:31:49.701383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701059 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701062 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701066 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701069 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701072 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701075 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701077 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701080 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701084 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701087 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701090 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701092 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701095 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701098 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701103 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701106 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701109 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701112 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701115 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:31:49.701855 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701118 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701121 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701124 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701127 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701129 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701132 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701135 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701137 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701140 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701143 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701145 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701148 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701151 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701154 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701156 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701160 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701163 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701165 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701168 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701172 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:31:49.702429 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701175 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701177 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701180 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701182 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701185 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701188 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.701190 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702096 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702107 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702111 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702114 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702116 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702119 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702122 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702125 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702128 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702131 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702133 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702136 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:31:49.702904 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702139 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702141 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702144 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702147 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702150 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702153 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702156 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702158 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702161 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702164 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702166 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702170 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702172 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702175 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702178 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702180 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702183 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702185 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702188 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702191 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:31:49.703416 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702194 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702197 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702200 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702204 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702208 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702211 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702214 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702217 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702220 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702223 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702225 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702228 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702231 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702234 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702236 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702239 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702242 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702244 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702247 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:31:49.703896 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702250 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702252 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702255 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702258 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702260 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702263 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702280 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702283 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702286 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702289 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702292 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702294 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702297 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702301 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702304 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702306 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702309 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702314 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702318 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:31:49.704436 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702322 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702325 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702328 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702331 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702333 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702336 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702340 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702343 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702345 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702348 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702350 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702353 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702355 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702358 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702361 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702364 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702366 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702369 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702372 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702375 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:31:49.704891 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702377 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702380 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702383 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702386 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702388 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702391 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702395 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702398 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702400 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702403 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702406 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702408 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702411 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702414 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702416 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702419 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702946 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702952 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702956 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:31:49.705383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702959 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702961 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702964 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702967 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702970 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702972 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702975 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702978 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702981 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702983 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702986 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702989 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702991 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702994 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702997 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.702999 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703002 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703005 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703008 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:31:49.705831 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703011 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703013 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703016 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703019 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703022 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703024 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703027 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703030 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703032 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703035 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703038 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703040 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703043 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703046 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703049 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703052 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703054 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703057 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703059 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703062 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:31:49.706303 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703065 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703067 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703069 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703072 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703074 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703077 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703080 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703082 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703085 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703087 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703090 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703094 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703097 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703099 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703102 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703105 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703108 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703110 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703113 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:31:49.706774 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703115 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703117 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703120 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703122 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703125 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703127 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703130 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703133 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703137 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703139 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703142 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703144 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703147 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703149 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703152 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703154 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703157 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703159 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703162 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703165 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:31:49.707220 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703167 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703171 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703175 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703179 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703183 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703186 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703189 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703191 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703194 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703196 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703199 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703201 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703204 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703206 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703209 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703212 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703214 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703217 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703219 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703222 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:31:49.707719 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703225 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703233 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703237 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703241 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.703244 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704362 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704375 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704389 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704394 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704399 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704402 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704407 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704413 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704416 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704420 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704423 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704427 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704430 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704434 2575 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704437 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704440 2575 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704443 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704447 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:31:49.708184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704452 2575 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704455 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704458 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704461 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704465 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704469 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704472 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704475 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704479 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704482 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704487 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704491 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704494 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704497 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704501 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704505 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704508 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704511 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704514 2575 flags.go:64] FLAG: --enable-server="true" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704517 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704522 2575 flags.go:64] FLAG: --event-burst="100" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704526 2575 flags.go:64] FLAG: --event-qps="50" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704529 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704532 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:31:49.708733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704535 2575 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704540 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704543 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704546 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704549 2575 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704552 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704555 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704558 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704561 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704564 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704567 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704570 2575 flags.go:64] FLAG: --feature-gates="" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704574 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704577 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704581 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704584 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704588 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704591 2575 flags.go:64] FLAG: --help="false" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704595 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-193.ec2.internal" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704598 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704602 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704605 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704609 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:31:49.709359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704612 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704615 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704619 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704621 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704624 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704627 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704631 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704634 2575 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704637 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704640 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704643 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704646 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704649 2575 flags.go:64] FLAG: --lock-file="" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704652 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704655 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704658 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704665 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704668 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704671 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704674 2575 flags.go:64] FLAG: --logging-format="text" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704677 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704680 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704683 2575 flags.go:64] FLAG: --manifest-url="" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704686 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704691 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:31:49.709913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704695 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704699 2575 flags.go:64] FLAG: --max-pods="110" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704702 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704706 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704709 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704713 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704716 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704719 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704722 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704731 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704734 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704737 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704740 2575 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704744 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704746 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704749 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704752 2575 flags.go:64] FLAG: --port="10250" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704755 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704758 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eda2151ec4cb49c9" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704762 2575 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704765 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704768 2575 flags.go:64] FLAG: --register-node="true" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704771 2575 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704775 2575 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704778 2575 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704781 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:31:49.710545 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704783 2575 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704788 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704791 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704794 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704797 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704799 2575 flags.go:64] FLAG: --runonce="false" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704805 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704808 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704811 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704814 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704817 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704821 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704825 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704828 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704831 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704834 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704837 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704840 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704843 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704846 2575 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704849 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704855 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704858 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704861 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704866 2575 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:31:49.712089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704869 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704872 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704875 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704878 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704881 2575 flags.go:64] FLAG: --v="2" Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704886 2575 flags.go:64] FLAG: --version="false" Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704891 2575 flags.go:64] FLAG: --vmodule="" Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704896 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.704899 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705045 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705049 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705052 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705055 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705059 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705062 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705066 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705069 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705072 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705074 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705078 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705081 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705083 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:31:49.713173 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705086 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705088 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705091 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705094 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705097 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705100 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705102 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705105 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705108 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705110 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705113 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705115 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705118 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705121 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705124 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705128 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705132 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705135 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705138 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:31:49.714141 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705141 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705144 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705147 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705150 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705154 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705158 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705163 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705166 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705170 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705173 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705176 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705183 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705186 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705189 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705192 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705195 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705198 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705201 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:31:49.715005 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705204 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705207 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705210 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705213 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705215 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705218 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705221 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705223 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705226 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705228 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705231 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705234 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705236 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705239 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705241 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705244 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705247 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705249 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705253 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705256 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:31:49.715706 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705260 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705263 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705285 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705290 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705294 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705297 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705301 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705304 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705307 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705310 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705312 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705333 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705337 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705340 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705343 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705347 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705349 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705352 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705355 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705358 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:31:49.716433 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705360 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705363 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705365 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705368 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705371 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705374 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705376 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705379 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705381 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705384 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705389 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705392 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705396 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705399 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705401 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705404 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:31:49.717236 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.705438 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705485 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705490 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705494 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705497 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705500 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705503 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705505 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705508 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705511 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705514 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705516 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705519 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705522 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705524 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705527 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705530 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705532 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705535 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705537 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:31:49.717843 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705540 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705543 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705545 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705547 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705550 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705553 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705557 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705560 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705562 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705566 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705569 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705572 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705575 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705578 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705580 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705583 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705587 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705589 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705592 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705595 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:31:49.718491 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705598 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705600 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705603 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705606 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705609 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705612 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705614 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705617 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705620 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705622 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705625 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705629 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705634 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705637 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705640 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705642 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705645 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705647 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705651 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:31:49.718973 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705654 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705656 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705662 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705666 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705669 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705672 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705675 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705678 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705680 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705683 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705687 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705689 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705692 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705695 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705698 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705700 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705703 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705705 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:31:49.719512 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705708 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705711 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705714 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705716 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705719 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705721 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705724 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705727 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705729 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705732 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705734 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705737 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705740 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705744 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705746 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705749 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705753 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705755 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705758 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705761 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:31:49.719951 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705763 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705766 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705769 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705771 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705774 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705777 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705780 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705782 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705785 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.705789 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.713178 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.713205 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.713230 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.713239 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713357 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713367 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713374 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713379 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713384 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:31:49.720518 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713392 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713399 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713405 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713410 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713414 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713421 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713425 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713430 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713434 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713438 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713443 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713447 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713452 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713456 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713461 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713465 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713470 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713474 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713478 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713482 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:31:49.720968 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713486 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713490 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713497 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713504 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713511 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713516 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713520 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713524 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713529 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713533 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713538 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713542 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713546 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713550 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713554 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713559 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713563 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713567 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713571 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:31:49.721561 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713576 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713580 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713585 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713589 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713593 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713597 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713602 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713606 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713610 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713615 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713619 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713624 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713629 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713634 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713638 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713643 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713647 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713652 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713657 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713662 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:31:49.722535 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713666 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713671 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713675 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713680 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713684 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713689 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713694 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713698 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713702 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713706 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713710 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713715 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713719 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713723 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713727 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713732 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713736 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713740 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713744 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:31:49.723401 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713748 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713753 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713757 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713761 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713766 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713770 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713775 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713779 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713783 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713788 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713792 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713796 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713802 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713806 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713811 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713815 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713819 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713824 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713828 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713832 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:31:49.724142 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713836 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713841 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.713846 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.713892 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714149 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714157 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714162 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714167 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714172 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714177 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714182 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714187 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714192 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714197 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714202 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714206 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:31:49.724750 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714211 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714216 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714220 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714224 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714229 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714234 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714238 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714243 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714247 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714260 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714281 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714286 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714291 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714296 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714300 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714305 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714309 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714313 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714317 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714321 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:31:49.725383 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714326 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714331 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714335 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714339 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714344 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714348 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714352 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714357 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714364 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714371 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714376 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714380 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714384 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714389 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714393 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714397 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714401 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714407 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714412 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714416 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:31:49.726161 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714420 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714425 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714431 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714435 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714440 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714445 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714449 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714453 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714459 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714463 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714467 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714471 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714476 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714480 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714485 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714489 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714493 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714498 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714502 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714506 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:31:49.726919 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714511 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714515 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714520 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714524 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714528 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714533 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714537 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714542 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714546 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714552 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714556 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714561 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714566 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714570 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714574 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714579 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714584 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714589 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714593 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714598 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:31:49.727451 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714602 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714607 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714612 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714616 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714620 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714624 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714629 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714633 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714637 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714641 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714645 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714651 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714657 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714662 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.714705 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:31:49.727931 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714762 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714770 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714774 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714779 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714786 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714793 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714798 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714803 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714807 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714812 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714816 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714820 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714824 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714829 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714834 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714838 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714843 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714847 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714852 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:31:49.728342 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714856 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714861 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714865 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714869 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714873 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714877 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714882 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714886 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714890 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714894 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714898 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714903 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714907 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714911 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714915 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714919 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714924 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714928 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714932 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714937 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:31:49.728869 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714941 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714945 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714949 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714953 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714957 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714961 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714966 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714971 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714975 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714980 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714984 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714988 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.714995 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715000 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715007 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715011 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715014 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715018 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715023 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715027 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:31:49.729639 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715031 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715035 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715040 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715044 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715049 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715053 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715057 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715061 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715067 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715074 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715079 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715083 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715088 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715093 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715097 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715102 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715106 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715110 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715114 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:31:49.730187 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715119 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715124 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715129 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715133 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715138 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715142 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715146 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715152 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715156 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715161 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715165 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715169 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715174 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715177 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715182 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715186 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715192 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715196 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715200 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:31:49.730670 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715205 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715209 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715213 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715218 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715222 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715226 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715230 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715234 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:49.715238 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.715342 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.718210 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.719340 2575 server.go:1014] "Starting client certificate rotation" Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.719437 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.720299 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.722023 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:31:49.731118 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.724001 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:31:49.734397 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.734379 2575 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:31:49.738281 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.738246 2575 log.go:25] "Validated CRI v1 image API" Oct 30 12:31:49.739194 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.739165 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:31:49.741681 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.741655 2575 fs.go:135] Filesystem UUIDs: map[426bc9d2-8242-4677-a9d6-8316a32fca56:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d9ecd2b2-462d-47a8-a0bc-aca898527fa3:/dev/nvme0n1p3] Oct 30 12:31:49.741736 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.741680 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 12:31:49.747651 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.747517 2575 manager.go:217] Machine: {Timestamp:2026-10-30 12:31:49.74640646 +0000 UTC m=+0.329654798 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095466 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f441c8d90bb88daaa94aca0132be SystemUUID:ec24f441-c8d9-0bb8-8daa-a94aca0132be BootID:345f4e25-b980-422f-aec4-168548d2d0eb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:5e:29:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:5e:29:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:af:58:b7:95:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:31:49.747651 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.747646 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:31:49.747812 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.747735 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:31:49.749172 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.749145 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:31:49.749419 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.749174 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:31:49.749502 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.749428 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:31:49.749502 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.749440 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:31:49.749585 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.749498 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:31:49.749585 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.749531 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:31:49.750354 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.750341 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:31:49.754127 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.754110 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:31:49.754297 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.754286 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:31:49.754899 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.754888 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:31:49.754951 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.754906 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:31:49.754951 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.754922 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:31:49.754951 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.754936 2575 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:31:49.754951 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.754948 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:31:49.755359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.755345 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:31:49.757850 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.757823 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:31:49.762747 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.762734 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:31:49.763562 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.763550 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:31:49.763610 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.763580 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:31:49.763754 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.763743 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:31:49.763782 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.763765 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:31:49.763809 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.763793 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:31:49.763809 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.763803 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:31:49.763858 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.763812 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:31:49.763858 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.763818 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:31:49.764394 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.764384 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:31:49.764394 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.764394 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:31:49.764458 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.764402 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:31:49.764458 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.764408 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:31:49.764458 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.764417 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:31:49.764458 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.764425 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:31:49.764975 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.764965 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:31:49.764975 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.764975 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:31:49.768936 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.768920 2575 server.go:1263] "Started kubelet" Oct 30 12:31:49.769008 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.768972 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:31:49.769359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.769174 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:31:49.769464 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.769450 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:31:49.769916 ip-10-0-134-193 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:31:49.776488 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.773250 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:31:49.776488 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.775954 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:31:49.776975 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.776928 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 12:31:49.781054 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.781030 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:31:49.781937 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.781918 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:31:49.782068 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.782035 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.782632 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.782714 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.782731 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.782804 2575 factory.go:55] Registering systemd factory Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.782820 2575 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.782873 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.782879 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.783033 2575 factory.go:153] Registering CRI-O factory Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.783049 2575 factory.go:223] Registration of the crio container factory successfully Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.783112 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.783129 2575 factory.go:103] Registering Raw factory Oct 30 12:31:49.784057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.783144 2575 manager.go:1199] Started watching for new ooms in manager Oct 30 12:31:49.784689 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.784096 2575 manager.go:319] Starting recovery of all containers Oct 30 12:31:49.795469 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.795447 2575 manager.go:324] Recovery completed Oct 30 12:31:49.799964 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.799946 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:31:49.801990 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.801962 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:31:49.802073 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.802008 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:31:49.802073 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.802024 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:31:49.802694 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.802587 2575 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:31:49.802694 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.802601 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:31:49.802694 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.802623 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:31:49.802881 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.802868 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:31:49.804784 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.804767 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:31:49.804869 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.804797 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:31:49.804869 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.804807 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:31:49.804869 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.804817 2575 policy_none.go:50] "Start" Oct 30 12:31:49.804869 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.804833 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:31:49.804869 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.804846 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:31:49.806914 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.806900 2575 policy_none.go:44] "Start" Oct 30 12:31:49.819032 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.818814 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:31:49.819032 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.818861 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mfbx7" Oct 30 12:31:49.828619 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.828581 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mfbx7" Oct 30 12:31:49.830968 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.830904 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 12:31:49.831088 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.831043 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 12:31:49.831723 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.830846 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-193.ec2.internal.18e34e9c48eaf5b5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-193.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-193.ec2.internal,},FirstTimestamp:2026-10-30 12:31:49.768811957 +0000 UTC m=+0.352060294,LastTimestamp:2026-10-30 12:31:49.768811957 +0000 UTC m=+0.352060294,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-193.ec2.internal,}" Oct 30 12:31:49.842901 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.842879 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-193.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 12:31:49.843003 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.842904 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 12:31:49.843167 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.843134 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:31:49.849803 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.849787 2575 manager.go:341] "Starting Device Plugin manager" Oct 30 12:31:49.849900 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.849819 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:31:49.849900 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.849829 2575 server.go:87] "Starting device plugin registration server" Oct 30 12:31:49.850144 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.850128 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:31:49.850234 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.850144 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:31:49.850234 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.850225 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:31:49.850367 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.850330 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:31:49.850367 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.850339 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:31:49.851125 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.851105 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:31:49.851226 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.851155 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:49.861384 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.861363 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 12:31:49.861457 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.861364 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-193.ec2.internal.18e34e9c4ae53b27 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-193.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-193.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-193.ec2.internal,},FirstTimestamp:2026-10-30 12:31:49.801990951 +0000 UTC m=+0.385239293,LastTimestamp:2026-10-30 12:31:49.801990951 +0000 UTC m=+0.385239293,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-193.ec2.internal,}" Oct 30 12:31:49.946607 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.879311 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-193.ec2.internal.18e34e9c4ae5968a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-193.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-193.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-193.ec2.internal,},FirstTimestamp:2026-10-30 12:31:49.802014346 +0000 UTC m=+0.385262685,LastTimestamp:2026-10-30 12:31:49.802014346 +0000 UTC m=+0.385262685,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-193.ec2.internal,}" Oct 30 12:31:49.946607 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.906901 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-193.ec2.internal.18e34e9c4ae5d249 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-193.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-193.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-193.ec2.internal,},FirstTimestamp:2026-10-30 12:31:49.802029641 +0000 UTC m=+0.385277981,LastTimestamp:2026-10-30 12:31:49.802029641 +0000 UTC m=+0.385277981,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-193.ec2.internal,}" Oct 30 12:31:49.946607 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.924301 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:31:49.946607 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.925577 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:31:49.946607 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.925599 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:31:49.946607 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.925623 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:31:49.946607 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.925658 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 12:31:49.946607 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.927969 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-193.ec2.internal.18e34e9c4ae53b27\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-193.ec2.internal.18e34e9c4ae53b27 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-193.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-193.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-193.ec2.internal,},FirstTimestamp:2026-10-30 12:31:49.801990951 +0000 UTC m=+0.385239293,LastTimestamp:2026-10-30 12:31:49.804782139 +0000 UTC m=+0.388030477,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-193.ec2.internal,}" Oct 30 12:31:49.950247 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.950221 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:31:49.951279 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.951246 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:31:49.951347 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.951299 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:31:49.951347 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.951313 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:31:49.951347 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.951344 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-193.ec2.internal" Oct 30 12:31:49.972568 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.972395 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:31:49.978947 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:49.978928 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-193.ec2.internal" Oct 30 12:31:49.979027 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:49.978955 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-193.ec2.internal\": node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.023819 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.023784 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.025770 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.025750 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal"] Oct 30 12:31:50.025822 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.025811 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:31:50.026781 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.026764 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:31:50.026883 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.026800 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:31:50.026883 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.026814 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:31:50.028771 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.028757 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:31:50.028923 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.028909 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.028970 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.028939 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:31:50.029825 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.029811 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:31:50.029905 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.029826 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:31:50.029905 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.029838 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:31:50.029905 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.029852 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:31:50.030048 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.029852 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:31:50.030048 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.029931 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:31:50.031766 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.031752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.031840 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.031777 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:31:50.032492 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.032477 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:31:50.032609 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.032501 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:31:50.032609 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.032515 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:31:50.054208 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.054187 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-193.ec2.internal\" not found" node="ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.056766 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.056749 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-193.ec2.internal\" not found" node="ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.083953 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.083919 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.083953 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.083950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.084137 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.083977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1f1b01344f2c1cc7bc384545bbc0754-config\") pod \"kube-apiserver-proxy-ip-10-0-134-193.ec2.internal\" (UID: \"f1f1b01344f2c1cc7bc384545bbc0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.124514 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.124474 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.184090 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.184057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.184090 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.184085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.184336 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.184111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1f1b01344f2c1cc7bc384545bbc0754-config\") pod \"kube-apiserver-proxy-ip-10-0-134-193.ec2.internal\" (UID: \"f1f1b01344f2c1cc7bc384545bbc0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.184336 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.184189 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.184336 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.184235 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1f1b01344f2c1cc7bc384545bbc0754-config\") pod \"kube-apiserver-proxy-ip-10-0-134-193.ec2.internal\" (UID: \"f1f1b01344f2c1cc7bc384545bbc0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.184336 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.184298 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.224968 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.224907 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.325592 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.325564 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.357758 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.357713 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.359892 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.359872 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:31:50.426240 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.426212 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.526540 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.526465 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.626705 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.626673 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.721169 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.721135 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 12:31:50.721697 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.721363 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 12:31:50.727895 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.727870 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.828594 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.828514 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.928614 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:50.928587 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:50.949230 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.949188 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:26:50 +0000 UTC" deadline="2028-07-23 18:19:16.960294006 +0000 UTC" Oct 30 12:31:50.949230 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.949226 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15173h47m26.011072346s" Oct 30 12:31:50.966558 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.966537 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:31:50.967581 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.967560 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:31:50.967707 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.967592 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:31:50.967707 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:50.967603 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:31:51.028690 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.028660 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:51.031903 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:51.031869 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode707ecff82755bdfb3889f9c2a2af067.slice/crio-d149e2fb3ac56044cd991c4a991df3aa3a36e6cda1e1d80f2768fee0ccc96d90 WatchSource:0}: Error finding container d149e2fb3ac56044cd991c4a991df3aa3a36e6cda1e1d80f2768fee0ccc96d90: Status 404 returned error can't find the container with id d149e2fb3ac56044cd991c4a991df3aa3a36e6cda1e1d80f2768fee0ccc96d90 Oct 30 12:31:51.039093 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.039072 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:31:51.042002 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:51.041969 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f1b01344f2c1cc7bc384545bbc0754.slice/crio-8aedbb4e9a0fdbed6117f4cb28a87473625265160767c63a17bab2a3b6b445cf WatchSource:0}: Error finding container 8aedbb4e9a0fdbed6117f4cb28a87473625265160767c63a17bab2a3b6b445cf: Status 404 returned error can't find the container with id 8aedbb4e9a0fdbed6117f4cb28a87473625265160767c63a17bab2a3b6b445cf Oct 30 12:31:51.129194 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.129106 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:51.140515 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.140495 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:31:51.229489 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.229454 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:51.284026 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.283990 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:31:51.329780 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.329736 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:31:51.399761 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.399682 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:31:51.483348 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.483308 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:31:51.510424 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.510389 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:31:51.511314 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.511286 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:31:51.535409 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.535365 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:31:51.756710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.756618 2575 apiserver.go:52] "Watching apiserver" Oct 30 12:31:51.783356 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.783321 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 12:31:51.790745 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.790711 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:31:51.791076 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.791053 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-2j4ks","kube-system/global-pull-secret-syncer-8fl8b","kube-system/konnectivity-agent-d44rw","openshift-multus/multus-additional-cni-plugins-v9vxj","openshift-network-diagnostics/network-check-target-kjjj6","openshift-image-registry/node-ca-wsxsd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal","openshift-network-operator/iptables-alerter-852jn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f","openshift-dns/node-resolver-2bkbt","openshift-multus/multus-kwjsh","openshift-ovn-kubernetes/ovnkube-node-68xp7","kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7zvjh"] Oct 30 12:31:51.793680 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.793654 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:51.796036 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.796010 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.798529 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.798500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:31:51.803300 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.803277 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.805230 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.804920 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:51.805230 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.805022 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:31:51.807726 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.807705 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:51.810576 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.810555 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.811610 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.811577 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v9g2h\"" Oct 30 12:31:51.811716 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.811676 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:31:51.811956 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.811942 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:31:51.813100 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.813082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.814004 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.813987 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:31:51.814004 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.813996 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fkzfg\"" Oct 30 12:31:51.814148 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.814074 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:31:51.814148 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.814079 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:31:51.814148 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.814097 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:31:51.815745 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.815657 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:51.815745 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.815725 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:31:51.816168 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.816139 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:31:51.816255 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.816208 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56d7b\"" Oct 30 12:31:51.816255 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.816224 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:31:51.818248 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.818229 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:51.818378 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.818356 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:31:51.820717 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.820699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:51.823633 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.823615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.829937 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.829916 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5fvrw\"" Oct 30 12:31:51.831264 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.831247 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:31:51.831355 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.831314 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:31:51.831509 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.831493 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cgbt6\"" Oct 30 12:31:51.855959 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.855927 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dm66t" Oct 30 12:31:51.856131 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.855972 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:31:51.884090 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.884062 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:31:51.893214 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-host\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.893214 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-registration-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-systemd-units\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbk6q\" (UniqueName: \"kubernetes.io/projected/102a0155-f391-4c1e-98d7-eea36a51615e-kube-api-access-lbk6q\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cnibin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m58t\" (UniqueName: \"kubernetes.io/projected/fb075adb-3219-4e67-a61a-00a6a422eef7-kube-api-access-5m58t\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893357 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-dbus\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-daemon-config\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5ld\" (UniqueName: \"kubernetes.io/projected/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-kube-api-access-rc5ld\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-kubelet\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.893470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-conf\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-lib-modules\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-log-socket\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-netd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-env-overrides\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3354c62-d061-4f12-9c22-b21455da0faf-ovn-node-metrics-cert\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-konnectivity-ca\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-conf-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-run\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-script-lib\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-agent-certs\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-kubelet-config\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-netns\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-etc-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-ovn\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.893972 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-config\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893843 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cnibin\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crm9\" (UniqueName: \"kubernetes.io/projected/5f15c7b4-5807-4438-81c7-5a392eca8b2a-kube-api-access-6crm9\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-kubernetes\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-tuned\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e218ebda-caba-45d2-963a-77b96b0af90f-serviceca\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.893985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-hostroot\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-etc-kubernetes\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysconfig\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-var-lib-kubelet\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-system-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-sys-fs\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894111 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-multus-certs\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.894786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g68t4\" (UniqueName: \"kubernetes.io/projected/f2e3b448-9d0e-4141-a929-72732655d497-kube-api-access-g68t4\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-node-log\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e218ebda-caba-45d2-963a-77b96b0af90f-host\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2e3b448-9d0e-4141-a929-72732655d497-iptables-alerter-script\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2e3b448-9d0e-4141-a929-72732655d497-host-slash\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24b12750-62c7-478f-bea4-8aee763363f3-hosts-file\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894270 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-socket-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cni-binary-copy\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmkcs\" (UniqueName: \"kubernetes.io/projected/24b12750-62c7-478f-bea4-8aee763363f3-kube-api-access-jmkcs\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894420 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhwv\" (UniqueName: \"kubernetes.io/projected/e218ebda-caba-45d2-963a-77b96b0af90f-kube-api-access-7bhwv\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-device-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-system-cni-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.895606 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-sys\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-kubelet\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-slash\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-var-lib-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-k8s-cni-cncf-io\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-bin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-tmp\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2b7j\" (UniqueName: \"kubernetes.io/projected/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-kube-api-access-d2b7j\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-systemd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-bin\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894758 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-socket-dir-parent\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-netns\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894791 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-multus\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-modprobe-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.896371 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-systemd\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.897210 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/e3354c62-d061-4f12-9c22-b21455da0faf-kube-api-access-pqzgz\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.897210 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:51.897210 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.894912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b12750-62c7-478f-bea4-8aee763363f3-tmp-dir\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:51.920799 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.920754 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921040 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921057 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921094 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921145 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921151 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921197 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921209 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921233 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921243 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921257 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921260 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7nlfh\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921291 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921314 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921333 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921333 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxtxl\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921364 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k87jk\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921368 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:31:51.921420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921378 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:31:51.922089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921464 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:31:51.922089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.921512 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kq49t\"" Oct 30 12:31:51.931661 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.931599 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" event={"ID":"f1f1b01344f2c1cc7bc384545bbc0754","Type":"ContainerStarted","Data":"8aedbb4e9a0fdbed6117f4cb28a87473625265160767c63a17bab2a3b6b445cf"} Oct 30 12:31:51.933045 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.933004 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" event={"ID":"e707ecff82755bdfb3889f9c2a2af067","Type":"ContainerStarted","Data":"d149e2fb3ac56044cd991c4a991df3aa3a36e6cda1e1d80f2768fee0ccc96d90"} Oct 30 12:31:51.996028 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.995976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-bin\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.996028 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-socket-dir-parent\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.996273 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-netns\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.996273 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996129 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-bin\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.996273 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-multus\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.996273 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996200 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-netns\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.996273 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-socket-dir-parent\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.996273 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-modprobe-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.996273 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-systemd\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.996273 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996265 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-multus\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/e3354c62-d061-4f12-9c22-b21455da0faf-kube-api-access-pqzgz\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-systemd\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996352 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-modprobe-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b12750-62c7-478f-bea4-8aee763363f3-tmp-dir\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996409 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-host\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-registration-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-systemd-units\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbk6q\" (UniqueName: \"kubernetes.io/projected/102a0155-f391-4c1e-98d7-eea36a51615e-kube-api-access-lbk6q\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cnibin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m58t\" (UniqueName: \"kubernetes.io/projected/fb075adb-3219-4e67-a61a-00a6a422eef7-kube-api-access-5m58t\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-dbus\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996590 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-registration-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-daemon-config\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996670 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-host\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.996710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5ld\" (UniqueName: \"kubernetes.io/projected/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-kube-api-access-rc5ld\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-kubelet\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996786 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b12750-62c7-478f-bea4-8aee763363f3-tmp-dir\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-conf\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-lib-modules\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996903 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-log-socket\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-netd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996960 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-kubelet\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-env-overrides\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996958 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-systemd-units\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996980 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-dbus\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.996995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3354c62-d061-4f12-9c22-b21455da0faf-ovn-node-metrics-cert\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-konnectivity-ca\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-conf-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-run\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.997569 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.997087 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-script-lib\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-agent-certs\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.997165 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret podName:ecc1b4c7-ef74-4105-b613-2f53486e7aad nodeName:}" failed. No retries permitted until 2026-10-30 12:31:52.497139438 +0000 UTC m=+3.080387787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret") pod "global-pull-secret-syncer-8fl8b" (UID: "ecc1b4c7-ef74-4105-b613-2f53486e7aad") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-kubelet-config\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-conf\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997272 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-netns\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-netns\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-etc-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-ovn\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-config\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cnibin\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crm9\" (UniqueName: \"kubernetes.io/projected/5f15c7b4-5807-4438-81c7-5a392eca8b2a-kube-api-access-6crm9\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997469 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997516 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-kubernetes\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997558 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-etc-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997566 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.998402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997596 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-ovn\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997679 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cnibin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997737 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-run\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997782 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-conf-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997827 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-daemon-config\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997830 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-netd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997870 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-log-socket\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998078 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-konnectivity-ca\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998079 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-kubelet-config\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998152 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cnibin\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-kubernetes\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.997104 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-lib-modules\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998395 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-tuned\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e218ebda-caba-45d2-963a-77b96b0af90f-serviceca\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-hostroot\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-etc-kubernetes\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:51.999201 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysconfig\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-var-lib-kubelet\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998616 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-hostroot\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-system-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-sys-fs\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998680 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-multus-certs\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g68t4\" (UniqueName: \"kubernetes.io/projected/f2e3b448-9d0e-4141-a929-72732655d497-kube-api-access-g68t4\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998736 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-multus-certs\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-node-log\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e218ebda-caba-45d2-963a-77b96b0af90f-host\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998789 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-etc-kubernetes\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2e3b448-9d0e-4141-a929-72732655d497-iptables-alerter-script\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2e3b448-9d0e-4141-a929-72732655d497-host-slash\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998871 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-config\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.000188 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998901 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-var-lib-kubelet\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999043 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysconfig\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.999147 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:51.999203 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs podName:102a0155-f391-4c1e-98d7-eea36a51615e nodeName:}" failed. No retries permitted until 2026-10-30 12:31:52.499185046 +0000 UTC m=+3.082433388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs") pod "network-metrics-daemon-2j4ks" (UID: "102a0155-f391-4c1e-98d7-eea36a51615e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999250 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-node-log\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999325 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e218ebda-caba-45d2-963a-77b96b0af90f-host\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999337 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999384 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-system-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999437 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-sys-fs\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999487 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2e3b448-9d0e-4141-a929-72732655d497-host-slash\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999529 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2e3b448-9d0e-4141-a929-72732655d497-iptables-alerter-script\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.998873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24b12750-62c7-478f-bea4-8aee763363f3-hosts-file\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-socket-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999675 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-env-overrides\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.001089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cni-binary-copy\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999749 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24b12750-62c7-478f-bea4-8aee763363f3-hosts-file\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkcs\" (UniqueName: \"kubernetes.io/projected/24b12750-62c7-478f-bea4-8aee763363f3-kube-api-access-jmkcs\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhwv\" (UniqueName: \"kubernetes.io/projected/e218ebda-caba-45d2-963a-77b96b0af90f-kube-api-access-7bhwv\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-device-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-system-cni-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999868 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-sys\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-kubelet\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-slash\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-var-lib-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:51.999997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-k8s-cni-cncf-io\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-bin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-tmp\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.002015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000135 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2b7j\" (UniqueName: \"kubernetes.io/projected/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-kube-api-access-d2b7j\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-systemd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000201 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-kubelet\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000294 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-device-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000339 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-system-cni-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000380 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000495 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000552 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-sys\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000871 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-script-lib\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000937 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-k8s-cni-cncf-io\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.000982 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-slash\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.001008 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cni-binary-copy\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.001034 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-var-lib-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.001221 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.001266 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-bin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.002743 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.001301 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-systemd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.003331 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.001724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-socket-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.003331 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.001750 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:52.003331 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.001980 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e218ebda-caba-45d2-963a-77b96b0af90f-serviceca\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:52.003331 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.002587 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-agent-certs\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:31:52.003331 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.002697 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3354c62-d061-4f12-9c22-b21455da0faf-ovn-node-metrics-cert\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.003756 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.003725 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-tuned\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.003811 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.003764 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-tmp\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.032472 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.032395 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/e3354c62-d061-4f12-9c22-b21455da0faf-kube-api-access-pqzgz\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.036891 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.036859 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:31:52.036891 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.036890 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:31:52.037136 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.036906 2575 projected.go:196] Error preparing data for projected volume kube-api-access-bt5zc for pod openshift-network-diagnostics/network-check-target-kjjj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:52.037136 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.037040 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc podName:88627b5e-75be-4029-ba55-51759aedc92d nodeName:}" failed. No retries permitted until 2026-10-30 12:31:52.537015716 +0000 UTC m=+3.120264066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bt5zc" (UniqueName: "kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc") pod "network-check-target-kjjj6" (UID: "88627b5e-75be-4029-ba55-51759aedc92d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:52.038459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.038432 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m58t\" (UniqueName: \"kubernetes.io/projected/fb075adb-3219-4e67-a61a-00a6a422eef7-kube-api-access-5m58t\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.039717 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.039689 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g68t4\" (UniqueName: \"kubernetes.io/projected/f2e3b448-9d0e-4141-a929-72732655d497-kube-api-access-g68t4\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:52.042428 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.042338 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5ld\" (UniqueName: \"kubernetes.io/projected/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-kube-api-access-rc5ld\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.043218 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.042994 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crm9\" (UniqueName: \"kubernetes.io/projected/5f15c7b4-5807-4438-81c7-5a392eca8b2a-kube-api-access-6crm9\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:52.043218 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.043183 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhwv\" (UniqueName: \"kubernetes.io/projected/e218ebda-caba-45d2-963a-77b96b0af90f-kube-api-access-7bhwv\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:52.044051 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.043993 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkcs\" (UniqueName: \"kubernetes.io/projected/24b12750-62c7-478f-bea4-8aee763363f3-kube-api-access-jmkcs\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:52.045694 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.045664 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2b7j\" (UniqueName: \"kubernetes.io/projected/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-kube-api-access-d2b7j\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.046484 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.046466 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbk6q\" (UniqueName: \"kubernetes.io/projected/102a0155-f391-4c1e-98d7-eea36a51615e-kube-api-access-lbk6q\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:52.109870 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.109826 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:31:52.120048 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.120012 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwjsh" Oct 30 12:31:52.122072 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:52.122037 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b12750_62c7_478f_bea4_8aee763363f3.slice/crio-eb81b96d43e797d98f7600765712d20665be5ad2d50f77401d5129ebe09d8501 WatchSource:0}: Error finding container eb81b96d43e797d98f7600765712d20665be5ad2d50f77401d5129ebe09d8501: Status 404 returned error can't find the container with id eb81b96d43e797d98f7600765712d20665be5ad2d50f77401d5129ebe09d8501 Oct 30 12:31:52.129142 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.129117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:31:52.132084 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:52.132054 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a5fae8_0a72_4dcd_99b7_4cc8c7c30c38.slice/crio-10551bd805ee44c030ec2cc0cc61050728fde9102f459b12c1aac64fea0f818b WatchSource:0}: Error finding container 10551bd805ee44c030ec2cc0cc61050728fde9102f459b12c1aac64fea0f818b: Status 404 returned error can't find the container with id 10551bd805ee44c030ec2cc0cc61050728fde9102f459b12c1aac64fea0f818b Oct 30 12:31:52.140284 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:52.139504 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b35dcc8_6a2f_4cc9_a810_85ec056b5899.slice/crio-8dc19077288f4a9d95e9ee2d85b372c3c317e20239177e2863272fee2d42a83e WatchSource:0}: Error finding container 8dc19077288f4a9d95e9ee2d85b372c3c317e20239177e2863272fee2d42a83e: Status 404 returned error can't find the container with id 8dc19077288f4a9d95e9ee2d85b372c3c317e20239177e2863272fee2d42a83e Oct 30 12:31:52.153725 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.153676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:31:52.156162 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.156128 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:31:52.164483 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.164453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:31:52.167129 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.167065 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:31:52.167420 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:52.167389 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f15c7b4_5807_4438_81c7_5a392eca8b2a.slice/crio-05f5ed45a54acd8d5c1fa8f2b50fe8668f55272b0bbe72a2261538c618537a80 WatchSource:0}: Error finding container 05f5ed45a54acd8d5c1fa8f2b50fe8668f55272b0bbe72a2261538c618537a80: Status 404 returned error can't find the container with id 05f5ed45a54acd8d5c1fa8f2b50fe8668f55272b0bbe72a2261538c618537a80 Oct 30 12:31:52.169358 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:52.169213 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode218ebda_caba_45d2_963a_77b96b0af90f.slice/crio-b6a3b5bb45b25d684802403358e3ab0025a146f34f27ea79a25dea6fa1eb1f51 WatchSource:0}: Error finding container b6a3b5bb45b25d684802403358e3ab0025a146f34f27ea79a25dea6fa1eb1f51: Status 404 returned error can't find the container with id b6a3b5bb45b25d684802403358e3ab0025a146f34f27ea79a25dea6fa1eb1f51 Oct 30 12:31:52.178887 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:52.178851 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb075adb_3219_4e67_a61a_00a6a422eef7.slice/crio-8bc47e92b400edaf7ac8f478ee6f2b859598b22530f61ee07f922ca67769c5d3 WatchSource:0}: Error finding container 8bc47e92b400edaf7ac8f478ee6f2b859598b22530f61ee07f922ca67769c5d3: Status 404 returned error can't find the container with id 8bc47e92b400edaf7ac8f478ee6f2b859598b22530f61ee07f922ca67769c5d3 Oct 30 12:31:52.179824 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.179802 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:31:52.181764 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:52.181740 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46c98aa_50ec_4a0f_b2e9_f89f777a1bc0.slice/crio-7872669761cb4deec07ecdfe3ae90d264bd86a05fc0cebf4fc35f0b0dd2d32b3 WatchSource:0}: Error finding container 7872669761cb4deec07ecdfe3ae90d264bd86a05fc0cebf4fc35f0b0dd2d32b3: Status 404 returned error can't find the container with id 7872669761cb4deec07ecdfe3ae90d264bd86a05fc0cebf4fc35f0b0dd2d32b3 Oct 30 12:31:52.192155 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:31:52.192130 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e3b448_9d0e_4141_a929_72732655d497.slice/crio-c4a31801efaf1ac66875a037a1bc2e27d75b76501ad396656efcc28ea194fdc8 WatchSource:0}: Error finding container c4a31801efaf1ac66875a037a1bc2e27d75b76501ad396656efcc28ea194fdc8: Status 404 returned error can't find the container with id c4a31801efaf1ac66875a037a1bc2e27d75b76501ad396656efcc28ea194fdc8 Oct 30 12:31:52.208504 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.208231 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:31:52.214913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.214883 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:31:52.505221 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.505188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:52.505409 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.505253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:52.505409 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.505377 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:52.505409 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.505380 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:52.505585 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.505444 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs podName:102a0155-f391-4c1e-98d7-eea36a51615e nodeName:}" failed. No retries permitted until 2026-10-30 12:31:53.505424747 +0000 UTC m=+4.088673133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs") pod "network-metrics-daemon-2j4ks" (UID: "102a0155-f391-4c1e-98d7-eea36a51615e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:52.505585 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.505464 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret podName:ecc1b4c7-ef74-4105-b613-2f53486e7aad nodeName:}" failed. No retries permitted until 2026-10-30 12:31:53.505454381 +0000 UTC m=+4.088702708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret") pod "global-pull-secret-syncer-8fl8b" (UID: "ecc1b4c7-ef74-4105-b613-2f53486e7aad") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:52.606655 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.606618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:52.606924 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.606898 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:31:52.607011 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.606934 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:31:52.607011 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.606949 2575 projected.go:196] Error preparing data for projected volume kube-api-access-bt5zc for pod openshift-network-diagnostics/network-check-target-kjjj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:52.607116 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:52.607035 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc podName:88627b5e-75be-4029-ba55-51759aedc92d nodeName:}" failed. No retries permitted until 2026-10-30 12:31:53.607013065 +0000 UTC m=+4.190261418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt5zc" (UniqueName: "kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc") pod "network-check-target-kjjj6" (UID: "88627b5e-75be-4029-ba55-51759aedc92d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:52.860549 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.860483 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:26:52 +0000 UTC" deadline="2028-07-28 00:47:21.397624566 +0000 UTC" Oct 30 12:31:52.860549 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.860547 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15276h15m28.537084813s" Oct 30 12:31:52.962594 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.961864 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" event={"ID":"f1f1b01344f2c1cc7bc384545bbc0754","Type":"ContainerStarted","Data":"e0993b09ccded105128262e46b28dce042b3dea1f6d1a51b43f39285882cbe87"} Oct 30 12:31:52.966784 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.966753 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" event={"ID":"e3354c62-d061-4f12-9c22-b21455da0faf","Type":"ContainerStarted","Data":"c3e169ca1811e75f1bf7f490075a92fb2c451a96050cdc0f71386e4f2e19b79d"} Oct 30 12:31:52.976311 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.976118 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" event={"ID":"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0","Type":"ContainerStarted","Data":"7872669761cb4deec07ecdfe3ae90d264bd86a05fc0cebf4fc35f0b0dd2d32b3"} Oct 30 12:31:52.981368 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.981300 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wsxsd" event={"ID":"e218ebda-caba-45d2-963a-77b96b0af90f","Type":"ContainerStarted","Data":"b6a3b5bb45b25d684802403358e3ab0025a146f34f27ea79a25dea6fa1eb1f51"} Oct 30 12:31:52.991075 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.991009 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" event={"ID":"5f15c7b4-5807-4438-81c7-5a392eca8b2a","Type":"ContainerStarted","Data":"05f5ed45a54acd8d5c1fa8f2b50fe8668f55272b0bbe72a2261538c618537a80"} Oct 30 12:31:52.995318 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:52.995233 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-852jn" event={"ID":"f2e3b448-9d0e-4141-a929-72732655d497","Type":"ContainerStarted","Data":"c4a31801efaf1ac66875a037a1bc2e27d75b76501ad396656efcc28ea194fdc8"} Oct 30 12:31:53.005428 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.005304 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" podStartSLOduration=2.00528748 podStartE2EDuration="2.00528748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:53.00468661 +0000 UTC m=+3.587934962" watchObservedRunningTime="2026-10-30 12:31:53.00528748 +0000 UTC m=+3.588535832" Oct 30 12:31:53.007040 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.006449 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" event={"ID":"fb075adb-3219-4e67-a61a-00a6a422eef7","Type":"ContainerStarted","Data":"8bc47e92b400edaf7ac8f478ee6f2b859598b22530f61ee07f922ca67769c5d3"} Oct 30 12:31:53.023472 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.018430 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d44rw" event={"ID":"5b35dcc8-6a2f-4cc9-a810-85ec056b5899","Type":"ContainerStarted","Data":"8dc19077288f4a9d95e9ee2d85b372c3c317e20239177e2863272fee2d42a83e"} Oct 30 12:31:53.023472 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.020995 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwjsh" event={"ID":"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38","Type":"ContainerStarted","Data":"10551bd805ee44c030ec2cc0cc61050728fde9102f459b12c1aac64fea0f818b"} Oct 30 12:31:53.024753 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.024726 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2bkbt" event={"ID":"24b12750-62c7-478f-bea4-8aee763363f3","Type":"ContainerStarted","Data":"eb81b96d43e797d98f7600765712d20665be5ad2d50f77401d5129ebe09d8501"} Oct 30 12:31:53.518553 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.518511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:53.518814 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.518570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:53.518814 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.518685 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:53.518814 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.518734 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret podName:ecc1b4c7-ef74-4105-b613-2f53486e7aad nodeName:}" failed. No retries permitted until 2026-10-30 12:31:55.518721207 +0000 UTC m=+6.101969533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret") pod "global-pull-secret-syncer-8fl8b" (UID: "ecc1b4c7-ef74-4105-b613-2f53486e7aad") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:53.519206 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.519084 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:53.519206 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.519139 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs podName:102a0155-f391-4c1e-98d7-eea36a51615e nodeName:}" failed. No retries permitted until 2026-10-30 12:31:55.519122304 +0000 UTC m=+6.102370646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs") pod "network-metrics-daemon-2j4ks" (UID: "102a0155-f391-4c1e-98d7-eea36a51615e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:53.619715 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.619680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:53.619909 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.619885 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:31:53.619972 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.619911 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:31:53.619972 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.619926 2575 projected.go:196] Error preparing data for projected volume kube-api-access-bt5zc for pod openshift-network-diagnostics/network-check-target-kjjj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:53.620078 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.619995 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc podName:88627b5e-75be-4029-ba55-51759aedc92d nodeName:}" failed. No retries permitted until 2026-10-30 12:31:55.619975701 +0000 UTC m=+6.203224045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt5zc" (UniqueName: "kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc") pod "network-check-target-kjjj6" (UID: "88627b5e-75be-4029-ba55-51759aedc92d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:53.861400 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.861230 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:26:52 +0000 UTC" deadline="2028-08-04 10:44:21.695448432 +0000 UTC" Oct 30 12:31:53.861400 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.861313 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15454h12m27.834140507s" Oct 30 12:31:53.929420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.928486 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:53.929420 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.928599 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:31:53.929420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.928921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:53.929420 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.928989 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:31:53.929420 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:53.929061 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:53.929420 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:53.929129 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:31:54.036315 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:54.036201 2575 generic.go:356] "Generic (PLEG): container finished" podID="e707ecff82755bdfb3889f9c2a2af067" containerID="b16c01813740e3d093b3d0198934043a5eb0fa8d1b65f621ce1b7b5478b2ba18" exitCode=0 Oct 30 12:31:54.037280 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:54.037191 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" event={"ID":"e707ecff82755bdfb3889f9c2a2af067","Type":"ContainerDied","Data":"b16c01813740e3d093b3d0198934043a5eb0fa8d1b65f621ce1b7b5478b2ba18"} Oct 30 12:31:55.073570 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:55.073529 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" event={"ID":"e707ecff82755bdfb3889f9c2a2af067","Type":"ContainerStarted","Data":"7f0533e363a23595f81c93d3a7670e62f6cfe639dcdf7bae8b1103fe15c23ee6"} Oct 30 12:31:55.118235 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:55.117768 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" podStartSLOduration=4.117747573 podStartE2EDuration="4.117747573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:55.11755181 +0000 UTC m=+5.700800189" watchObservedRunningTime="2026-10-30 12:31:55.117747573 +0000 UTC m=+5.700995925" Oct 30 12:31:55.536290 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:55.536181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:55.536290 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:55.536268 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:55.536507 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.536482 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:55.536579 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.536558 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs podName:102a0155-f391-4c1e-98d7-eea36a51615e nodeName:}" failed. No retries permitted until 2026-10-30 12:31:59.536537682 +0000 UTC m=+10.119786027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs") pod "network-metrics-daemon-2j4ks" (UID: "102a0155-f391-4c1e-98d7-eea36a51615e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:55.537062 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.536651 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:55.537062 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.536697 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret podName:ecc1b4c7-ef74-4105-b613-2f53486e7aad nodeName:}" failed. No retries permitted until 2026-10-30 12:31:59.536683516 +0000 UTC m=+10.119931858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret") pod "global-pull-secret-syncer-8fl8b" (UID: "ecc1b4c7-ef74-4105-b613-2f53486e7aad") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:55.636724 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:55.636680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:55.636916 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.636882 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:31:55.636916 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.636903 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:31:55.637026 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.636918 2575 projected.go:196] Error preparing data for projected volume kube-api-access-bt5zc for pod openshift-network-diagnostics/network-check-target-kjjj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:55.637026 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.636979 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc podName:88627b5e-75be-4029-ba55-51759aedc92d nodeName:}" failed. No retries permitted until 2026-10-30 12:31:59.636961529 +0000 UTC m=+10.220209861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt5zc" (UniqueName: "kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc") pod "network-check-target-kjjj6" (UID: "88627b5e-75be-4029-ba55-51759aedc92d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:55.927556 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:55.927477 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:55.927556 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:55.927503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:55.927784 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.927626 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:31:55.928146 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.928117 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:31:55.928228 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:55.928201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:55.928317 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:55.928297 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:31:56.086557 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:56.086506 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="68eafaa16eb579e675e0199706aec52ca5896460f4695d631df5e66ec094660d" exitCode=0 Oct 30 12:31:56.087054 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:56.086661 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" event={"ID":"5f15c7b4-5807-4438-81c7-5a392eca8b2a","Type":"ContainerDied","Data":"68eafaa16eb579e675e0199706aec52ca5896460f4695d631df5e66ec094660d"} Oct 30 12:31:57.091696 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:57.091620 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d44rw" event={"ID":"5b35dcc8-6a2f-4cc9-a810-85ec056b5899","Type":"ContainerStarted","Data":"4e1dfca26df6d234b0ef14f753d5f4c3b4d7627b70d8bbc9d6b21da8b1400055"} Oct 30 12:31:57.928362 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:57.928319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:57.928510 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:57.928449 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:31:57.928854 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:57.928828 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:57.928948 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:57.928915 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:31:57.929015 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:57.928979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:57.929087 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:57.929059 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:31:58.099515 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:58.099479 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wsxsd" event={"ID":"e218ebda-caba-45d2-963a-77b96b0af90f","Type":"ContainerStarted","Data":"8a45973069705f9a4be406195dc8b89f2ec264e9f3c1e51fe5b8fc11bc311584"} Oct 30 12:31:58.141342 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:58.140794 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d44rw" podStartSLOduration=3.791444581 podStartE2EDuration="8.140780472s" totalImagesPullingTime="4.349335891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:57.133444679 +0000 UTC m=+7.716693030" watchObservedRunningTime="2026-10-30 12:31:58.140780472 +0000 UTC m=+8.724028886" Oct 30 12:31:59.105102 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.105071 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" event={"ID":"fb075adb-3219-4e67-a61a-00a6a422eef7","Type":"ContainerStarted","Data":"dd082559de0426b310933f98ad32f68acd9e084cf6084a16becfc93245528ec0"} Oct 30 12:31:59.569869 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.569829 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:59.570035 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.569915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:59.570092 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.570052 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:59.570144 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.570114 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret podName:ecc1b4c7-ef74-4105-b613-2f53486e7aad nodeName:}" failed. No retries permitted until 2026-10-30 12:32:07.570097506 +0000 UTC m=+18.153345831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret") pod "global-pull-secret-syncer-8fl8b" (UID: "ecc1b4c7-ef74-4105-b613-2f53486e7aad") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:31:59.570213 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.570185 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:59.570261 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.570220 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs podName:102a0155-f391-4c1e-98d7-eea36a51615e nodeName:}" failed. No retries permitted until 2026-10-30 12:32:07.570208802 +0000 UTC m=+18.153457144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs") pod "network-metrics-daemon-2j4ks" (UID: "102a0155-f391-4c1e-98d7-eea36a51615e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:31:59.642810 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.642720 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:31:59.671138 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.670762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:59.671138 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.670956 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:31:59.671138 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.670983 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:31:59.671138 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.670996 2575 projected.go:196] Error preparing data for projected volume kube-api-access-bt5zc for pod openshift-network-diagnostics/network-check-target-kjjj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:59.671138 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.671057 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc podName:88627b5e-75be-4029-ba55-51759aedc92d nodeName:}" failed. No retries permitted until 2026-10-30 12:32:07.671038591 +0000 UTC m=+18.254286922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt5zc" (UniqueName: "kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc") pod "network-check-target-kjjj6" (UID: "88627b5e-75be-4029-ba55-51759aedc92d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:31:59.855288 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.855186 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:31:59.64274119Z","UUID":"53968865-4f5a-4376-a203-592a4207a0a0","Handler":null,"Name":"","Endpoint":""} Oct 30 12:31:59.857422 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.857366 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:31:59.857422 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.857398 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:31:59.931516 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.931418 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:31:59.931676 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.931613 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:31:59.932130 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.932087 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:31:59.932246 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.932213 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:31:59.932388 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:31:59.932373 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:31:59.932562 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:31:59.932540 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:00.115033 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:00.114993 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" event={"ID":"fb075adb-3219-4e67-a61a-00a6a422eef7","Type":"ContainerStarted","Data":"469aa9bf900ccb64e192e238f2d8da3099bcf0ee454eafa81baf6e75dcb2d6a6"} Oct 30 12:32:00.159691 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:00.159659 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:32:00.160744 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:00.160720 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:32:00.220183 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:00.220061 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wsxsd" podStartSLOduration=5.212637256 podStartE2EDuration="10.220039235s" totalImagesPullingTime="5.007401979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:58.141442951 +0000 UTC m=+8.724691299" watchObservedRunningTime="2026-10-30 12:32:00.220039235 +0000 UTC m=+10.803287583" Oct 30 12:32:01.120771 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.120681 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" event={"ID":"fb075adb-3219-4e67-a61a-00a6a422eef7","Type":"ContainerStarted","Data":"dc19a1477c7c54914d215a83a646e5a840df3bf5571b0954ef3367a2b9d83c7a"} Oct 30 12:32:01.124092 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.124056 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwjsh" event={"ID":"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38","Type":"ContainerStarted","Data":"e0f9e712f278882259683ad5fbfcea331477e992987582ba96fc0aaaa9e2a568"} Oct 30 12:32:01.127213 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.126230 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2bkbt" event={"ID":"24b12750-62c7-478f-bea4-8aee763363f3","Type":"ContainerStarted","Data":"c1b29e9cfeda4059a931ee54f3d2ee6c5c7e51a4675a9ff57c38a9df7cf2b384"} Oct 30 12:32:01.129162 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.129131 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="a1db3ebb83f75eebfefe37bcb00f63ca31ca102e376d0a5128a32fa9017da99c" exitCode=0 Oct 30 12:32:01.129351 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.129330 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" event={"ID":"5f15c7b4-5807-4438-81c7-5a392eca8b2a","Type":"ContainerDied","Data":"a1db3ebb83f75eebfefe37bcb00f63ca31ca102e376d0a5128a32fa9017da99c"} Oct 30 12:32:01.168132 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.168080 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" podStartSLOduration=3.039217092 podStartE2EDuration="11.168066496s" totalImagesPullingTime="8.128849404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:01.167726477 +0000 UTC m=+11.750974827" watchObservedRunningTime="2026-10-30 12:32:01.168066496 +0000 UTC m=+11.751314842" Oct 30 12:32:01.264400 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.264343 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2bkbt" podStartSLOduration=3.310120294 podStartE2EDuration="11.264326787s" totalImagesPullingTime="7.954206493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:01.264291655 +0000 UTC m=+11.847540003" watchObservedRunningTime="2026-10-30 12:32:01.264326787 +0000 UTC m=+11.847575136" Oct 30 12:32:01.929422 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.929393 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:01.929422 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.929410 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:01.929643 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:01.929546 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:01.929643 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:01.929590 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:01.929737 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:01.929692 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:01.929803 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:01.929776 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:02.134440 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:02.134403 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="bc14715900b2d796449387e601a3dd97016d4b68c868d9e30f0419677fb995c7" exitCode=0 Oct 30 12:32:02.134885 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:02.134477 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" event={"ID":"5f15c7b4-5807-4438-81c7-5a392eca8b2a","Type":"ContainerDied","Data":"bc14715900b2d796449387e601a3dd97016d4b68c868d9e30f0419677fb995c7"} Oct 30 12:32:02.202955 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:02.202899 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kwjsh" podStartSLOduration=4.280019807 podStartE2EDuration="12.202884285s" totalImagesPullingTime="7.922864478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:01.312079393 +0000 UTC m=+11.895327740" watchObservedRunningTime="2026-10-30 12:32:02.202884285 +0000 UTC m=+12.786132631" Oct 30 12:32:03.139170 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:03.139129 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" event={"ID":"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0","Type":"ContainerStarted","Data":"dafd3881484dfaff8ef88695b305ae787373d97ed4a170a4509c553d547126b2"} Oct 30 12:32:03.142152 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:03.142054 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="52761aec52d8ec2d1cf12981554224e152420cd8696d43dfc28e14145aa493ab" exitCode=0 Oct 30 12:32:03.142152 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:03.142142 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" event={"ID":"5f15c7b4-5807-4438-81c7-5a392eca8b2a","Type":"ContainerDied","Data":"52761aec52d8ec2d1cf12981554224e152420cd8696d43dfc28e14145aa493ab"} Oct 30 12:32:03.144241 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:03.144221 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-852jn" event={"ID":"f2e3b448-9d0e-4141-a929-72732655d497","Type":"ContainerStarted","Data":"d7163933ff1078ad9bc7dee732c4e183904fcfd7ff82ea8be751ec87ea304f7b"} Oct 30 12:32:03.185811 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:03.185745 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" podStartSLOduration=2.770661835 podStartE2EDuration="13.185728971s" totalImagesPullingTime="10.415067136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:03.185094331 +0000 UTC m=+13.768342679" watchObservedRunningTime="2026-10-30 12:32:03.185728971 +0000 UTC m=+13.768977315" Oct 30 12:32:03.227092 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:03.226995 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-852jn" podStartSLOduration=4.889513566 podStartE2EDuration="13.226977811s" totalImagesPullingTime="8.337464245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:03.226863008 +0000 UTC m=+13.810111355" watchObservedRunningTime="2026-10-30 12:32:03.226977811 +0000 UTC m=+13.810226160" Oct 30 12:32:03.937202 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:03.937165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:03.937398 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:03.937344 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:03.937613 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:03.937593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:03.937830 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:03.937778 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:03.938357 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:03.937661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:03.938469 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:03.938445 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:04.577672 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:04.577586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:04.830294 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:04.830000 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:05.151412 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:05.151378 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="e4fbf7e79aea63c2f98239bf266f2330c80f64c8410fa3ed9b90f8d1a50de61f" exitCode=0 Oct 30 12:32:05.151570 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:05.151430 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" event={"ID":"5f15c7b4-5807-4438-81c7-5a392eca8b2a","Type":"ContainerDied","Data":"e4fbf7e79aea63c2f98239bf266f2330c80f64c8410fa3ed9b90f8d1a50de61f"} Oct 30 12:32:05.926666 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:05.926623 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:05.927102 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:05.926626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:05.927102 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:05.926759 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:05.927102 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:05.926856 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:05.927102 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:05.926623 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:05.927102 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:05.926957 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:06.160900 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:06.160864 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="bc8d2b2d7e5cbfe6b4eca891ae926062498450f5fee4c0b737a3ad20f32a49c3" exitCode=0 Oct 30 12:32:06.161070 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:06.160906 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" event={"ID":"5f15c7b4-5807-4438-81c7-5a392eca8b2a","Type":"ContainerDied","Data":"bc8d2b2d7e5cbfe6b4eca891ae926062498450f5fee4c0b737a3ad20f32a49c3"} Oct 30 12:32:07.168823 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.168633 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" event={"ID":"e3354c62-d061-4f12-9c22-b21455da0faf","Type":"ContainerStarted","Data":"71327ed34bddaca22d26127f180bd7d2f1f553c66acb85e95a4ef20b6854d738"} Oct 30 12:32:07.169284 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.168912 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" event={"ID":"e3354c62-d061-4f12-9c22-b21455da0faf","Type":"ContainerStarted","Data":"0d303c3d119d5c16da36903ee0a21fcd8aca3cbce562e1d4f006b3c9c08ee105"} Oct 30 12:32:07.169284 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.168951 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" event={"ID":"e3354c62-d061-4f12-9c22-b21455da0faf","Type":"ContainerStarted","Data":"0e715892b0b1caad54dbe9adb9cad655d1411f5288d30d71bc3cac28fa953db7"} Oct 30 12:32:07.169284 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.168994 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" event={"ID":"e3354c62-d061-4f12-9c22-b21455da0faf","Type":"ContainerStarted","Data":"8f390cec0e020df36362a5e0752da0d0717617ff0a8a783c1e43dc34c50a8af0"} Oct 30 12:32:07.169284 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.169021 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" event={"ID":"e3354c62-d061-4f12-9c22-b21455da0faf","Type":"ContainerStarted","Data":"206e25cc344e9c6100c9eea102fa8145a622c7dc08d7f57998f9d8b266ded496"} Oct 30 12:32:07.169284 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.169046 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" event={"ID":"e3354c62-d061-4f12-9c22-b21455da0faf","Type":"ContainerStarted","Data":"39365079d324da81de3d3bb6d3107c0bca3889436a63df70d4cd2e332d9d32a8"} Oct 30 12:32:07.171806 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.171777 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" event={"ID":"5f15c7b4-5807-4438-81c7-5a392eca8b2a","Type":"ContainerStarted","Data":"823e1e8de4502bf93f96aeb2fcaf612cad30a73f15f92ec7a846706cb597f058"} Oct 30 12:32:07.210905 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.210875 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:07.414772 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.414740 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:07.635316 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.635290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:07.635457 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.635346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:07.635457 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.635429 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:32:07.635525 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.635480 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret podName:ecc1b4c7-ef74-4105-b613-2f53486e7aad nodeName:}" failed. No retries permitted until 2026-10-30 12:32:23.635468719 +0000 UTC m=+34.218717044 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret") pod "global-pull-secret-syncer-8fl8b" (UID: "ecc1b4c7-ef74-4105-b613-2f53486e7aad") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:32:07.635525 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.635430 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:32:07.635594 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.635554 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs podName:102a0155-f391-4c1e-98d7-eea36a51615e nodeName:}" failed. No retries permitted until 2026-10-30 12:32:23.635539614 +0000 UTC m=+34.218787959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs") pod "network-metrics-daemon-2j4ks" (UID: "102a0155-f391-4c1e-98d7-eea36a51615e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:32:07.735816 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.735783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:07.735995 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.735897 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:32:07.735995 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.735914 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:32:07.735995 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.735923 2575 projected.go:196] Error preparing data for projected volume kube-api-access-bt5zc for pod openshift-network-diagnostics/network-check-target-kjjj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:32:07.735995 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.735968 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc podName:88627b5e-75be-4029-ba55-51759aedc92d nodeName:}" failed. No retries permitted until 2026-10-30 12:32:23.735956748 +0000 UTC m=+34.319205073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt5zc" (UniqueName: "kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc") pod "network-check-target-kjjj6" (UID: "88627b5e-75be-4029-ba55-51759aedc92d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:32:07.799647 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.799611 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:07.926208 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.926124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:07.926208 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.926153 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:07.926413 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.926316 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:07.926413 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:07.926357 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:07.926507 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.926464 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:07.926587 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:07.926566 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:08.004862 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:08.004833 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:09.179786 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:09.179754 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" event={"ID":"e3354c62-d061-4f12-9c22-b21455da0faf","Type":"ContainerStarted","Data":"9945ad68f4fc398800dfc560aa6842d5fd0456919c2f0c2be65ad50de4dc6d8d"} Oct 30 12:32:09.926877 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:09.926842 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:09.927053 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:09.926968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:09.927053 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:09.926976 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:09.927139 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:09.927062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:09.927177 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:09.927060 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:09.927177 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:09.927143 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:11.083792 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.083763 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:11.191140 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.191109 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" event={"ID":"e3354c62-d061-4f12-9c22-b21455da0faf","Type":"ContainerStarted","Data":"fc78e2f74d4661848dfe4eec5162f7a2bd2b2892c1f0ffc7f3a6978b812932d8"} Oct 30 12:32:11.191465 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.191449 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:32:11.207779 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.207600 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:32:11.300673 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.300554 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" podStartSLOduration=6.896100934 podStartE2EDuration="21.300541354s" totalImagesPullingTime="14.40444042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:11.300106322 +0000 UTC m=+21.883354670" watchObservedRunningTime="2026-10-30 12:32:11.300541354 +0000 UTC m=+21.883789700" Oct 30 12:32:11.300886 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.300864 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v9vxj" podStartSLOduration=11.025236633 podStartE2EDuration="21.300858296s" totalImagesPullingTime="10.275621663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:07.225613796 +0000 UTC m=+17.808862142" watchObservedRunningTime="2026-10-30 12:32:11.300858296 +0000 UTC m=+21.884106642" Oct 30 12:32:11.310660 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.310631 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:11.545499 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.545469 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:11.738901 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.738866 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:32:11.739090 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.739034 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:32:11.739429 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.739412 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:32:11.784310 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.784260 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:11.926787 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.926754 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:11.926787 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.926793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:11.926976 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:11.926873 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:11.927015 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:11.926958 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:11.927049 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:11.927014 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:11.927110 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:11.927094 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:12.043981 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:12.043909 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:12.193950 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:12.193918 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:32:12.194388 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:12.193992 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:32:12.208543 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:12.208508 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:32:12.415967 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:12.415920 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:12.655653 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:12.655624 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:12.882748 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:12.882718 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:13.191522 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:13.191440 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8fl8b"] Oct 30 12:32:13.191699 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:13.191550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:13.191699 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:13.191632 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:13.195046 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:13.194736 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2j4ks"] Oct 30 12:32:13.195046 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:13.194856 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:13.195046 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:13.194985 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:13.195493 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:13.195151 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjjj6"] Oct 30 12:32:13.195493 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:13.195237 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:13.195493 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:13.195354 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:13.209569 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:13.209530 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:13.656066 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:13.656035 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:14.798657 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:14.798628 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2bkbt_24b12750-62c7-478f-bea4-8aee763363f3/dns-node-resolver/0.log" Oct 30 12:32:14.925974 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:14.925946 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:14.925974 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:14.925982 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:14.926181 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:14.925986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:14.926181 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:14.926058 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:14.926181 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:14.926107 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:14.926350 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:14.926202 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:16.198565 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:16.198539 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wsxsd_e218ebda-caba-45d2-963a-77b96b0af90f/node-ca/0.log" Oct 30 12:32:16.245624 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:16.245590 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:16.926510 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:16.926482 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:16.926510 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:16.926507 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:16.926803 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:16.926482 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:16.926803 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:16.926605 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:16.926803 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:16.926659 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:16.926803 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:16.926751 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:17.175406 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:17.175373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:17.396841 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:17.396803 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:17.597106 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:17.597073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:17.804688 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:17.804648 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:18.816096 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:18.815988 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:18.925929 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:18.925890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:18.926099 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:18.925955 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:18.926099 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:18.926052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:18.926099 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:18.926068 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:18.926208 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:18.926139 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:18.926251 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:18.926227 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:19.020646 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:19.020607 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:19.218869 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:19.218835 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:19.463181 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:19.463148 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:19.672326 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:19.672292 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:20.307727 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:20.307690 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:20.527488 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:20.527452 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:20.737214 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:20.737174 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:20.926695 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:20.926645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:20.926695 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:20.926698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:20.926910 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:20.926698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:20.926910 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:20.926785 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:20.926910 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:20.926850 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:20.927005 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:20.926933 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:20.939410 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:20.939378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:21.142663 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:21.142629 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:21.355148 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:21.355118 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:21.567184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:21.567109 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:21.774023 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:21.773993 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:22.221187 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:22.221154 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:22.439225 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:22.439193 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:22.810002 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:22.809969 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:22.926353 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:22.926327 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:22.926510 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:22.926334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:22.926510 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:22.926448 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:22.926510 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:22.926443 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:22.926677 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:22.926522 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:22.926677 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:22.926584 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:23.019012 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:23.018981 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:23.232258 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:23.232217 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:23.646911 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:23.646879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:23.647355 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:23.646928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:23.647355 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:23.647039 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:32:23.647355 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:23.647046 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:32:23.647355 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:23.647094 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs podName:102a0155-f391-4c1e-98d7-eea36a51615e nodeName:}" failed. No retries permitted until 2026-10-30 12:32:55.647077822 +0000 UTC m=+66.230326149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs") pod "network-metrics-daemon-2j4ks" (UID: "102a0155-f391-4c1e-98d7-eea36a51615e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:32:23.647355 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:23.647108 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret podName:ecc1b4c7-ef74-4105-b613-2f53486e7aad nodeName:}" failed. No retries permitted until 2026-10-30 12:32:55.647102052 +0000 UTC m=+66.230350377 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret") pod "global-pull-secret-syncer-8fl8b" (UID: "ecc1b4c7-ef74-4105-b613-2f53486e7aad") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:32:23.710313 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:23.710280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:23.747175 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:23.747148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:23.747314 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:23.747304 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:32:23.747357 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:23.747322 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:32:23.747357 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:23.747332 2575 projected.go:196] Error preparing data for projected volume kube-api-access-bt5zc for pod openshift-network-diagnostics/network-check-target-kjjj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:32:23.747419 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:23.747374 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc podName:88627b5e-75be-4029-ba55-51759aedc92d nodeName:}" failed. No retries permitted until 2026-10-30 12:32:55.747362677 +0000 UTC m=+66.330611015 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bt5zc" (UniqueName: "kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc") pod "network-check-target-kjjj6" (UID: "88627b5e-75be-4029-ba55-51759aedc92d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:32:23.943145 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:23.943066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:24.810314 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:24.810262 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:24.926192 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:24.926158 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:24.926192 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:24.926179 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:24.926428 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:24.926207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:24.926428 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:24.926311 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:24.926428 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:24.926355 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:24.926523 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:24.926429 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:25.012917 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:25.012884 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:25.223044 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:25.223010 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:25.430694 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:25.430659 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:25.637749 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:25.637714 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:25.974689 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:25.974602 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:26.815433 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:26.815400 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:26.926925 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:26.926885 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:26.927090 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:26.926900 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:26.927090 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:26.927010 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:26.927090 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:26.927007 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:26.927188 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:26.927107 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:26.927188 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:26.927156 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:27.030392 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:27.030362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:27.242374 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:27.242341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:27.444553 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:27.444515 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:27.655423 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:27.655392 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:27.855434 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:27.855403 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:28.052231 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:28.052153 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:28.258716 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:28.258685 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:28.926430 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:28.926391 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:28.926430 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:28.926423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:28.926675 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:28.926496 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:28.926675 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:28.926561 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:28.926782 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:28.926670 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:28.926782 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:28.926755 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:30.926426 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:30.926394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:30.926426 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:30.926423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:30.926911 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:30.926482 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjjj6" podUID="88627b5e-75be-4029-ba55-51759aedc92d" Oct 30 12:32:30.926911 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:30.926546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:30.926911 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:30.926616 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8fl8b" podUID="ecc1b4c7-ef74-4105-b613-2f53486e7aad" Oct 30 12:32:30.926911 ip-10-0-134-193 kubenswrapper[2575]: E1030 12:32:30.926699 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2j4ks" podUID="102a0155-f391-4c1e-98d7-eea36a51615e" Oct 30 12:32:31.363608 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.363538 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:31.583796 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.583766 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:31.634366 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.634343 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeReady" Oct 30 12:32:31.634513 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.634463 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 12:32:31.707359 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.707260 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dxl2t"] Oct 30 12:32:31.725463 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.725433 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fqntp"] Oct 30 12:32:31.725618 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.725564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:31.735542 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.735519 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8njh7"] Oct 30 12:32:31.735683 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.735666 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:32:31.741787 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.741764 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fqntp"] Oct 30 12:32:31.741909 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.741808 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dxl2t"] Oct 30 12:32:31.741909 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.741904 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:31.742499 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.742436 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:32:31.742818 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.742731 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5lgc6\"" Oct 30 12:32:31.742890 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.742829 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:32:31.753431 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.753410 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:32:31.773848 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.773824 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8njh7"] Oct 30 12:32:31.843057 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.843027 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:32:31.843247 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.843093 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:32:31.843247 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.843112 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bklll\"" Oct 30 12:32:31.843247 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.843150 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:32:31.843247 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.843183 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7rclv\"" Oct 30 12:32:31.843247 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.843191 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:32:31.843470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.843291 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:32:31.843470 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.843300 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:32:31.904419 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f156b-c309-4b4b-8a77-270faeb79648-data-volume\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:31.904419 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904369 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc939343-5581-480f-8c67-37ba0d1188c6-tmp-dir\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:31.904621 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f156b-c309-4b4b-8a77-270faeb79648-crio-socket\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:31.904621 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0686737a-10d7-4761-877c-46bd3df261c2-cert\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:32:31.904621 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcs5g\" (UniqueName: \"kubernetes.io/projected/0686737a-10d7-4761-877c-46bd3df261c2-kube-api-access-vcs5g\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:32:31.904733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f156b-c309-4b4b-8a77-270faeb79648-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:31.904733 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc939343-5581-480f-8c67-37ba0d1188c6-metrics-tls\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:31.904800 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f156b-c309-4b4b-8a77-270faeb79648-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:31.904800 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpc4l\" (UniqueName: \"kubernetes.io/projected/db5f156b-c309-4b4b-8a77-270faeb79648-kube-api-access-cpc4l\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:31.904884 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5l8w\" (UniqueName: \"kubernetes.io/projected/dc939343-5581-480f-8c67-37ba0d1188c6-kube-api-access-q5l8w\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:31.904884 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:31.904823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc939343-5581-480f-8c67-37ba0d1188c6-config-volume\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.005809 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.005778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f156b-c309-4b4b-8a77-270faeb79648-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.005809 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.005809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc939343-5581-480f-8c67-37ba0d1188c6-metrics-tls\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.006337 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.005840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f156b-c309-4b4b-8a77-270faeb79648-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.006337 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpc4l\" (UniqueName: \"kubernetes.io/projected/db5f156b-c309-4b4b-8a77-270faeb79648-kube-api-access-cpc4l\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.006337 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5l8w\" (UniqueName: \"kubernetes.io/projected/dc939343-5581-480f-8c67-37ba0d1188c6-kube-api-access-q5l8w\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.006337 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc939343-5581-480f-8c67-37ba0d1188c6-config-volume\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.006337 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f156b-c309-4b4b-8a77-270faeb79648-data-volume\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.006337 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc939343-5581-480f-8c67-37ba0d1188c6-tmp-dir\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.006337 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f156b-c309-4b4b-8a77-270faeb79648-crio-socket\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.006337 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0686737a-10d7-4761-877c-46bd3df261c2-cert\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:32:32.006337 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs5g\" (UniqueName: \"kubernetes.io/projected/0686737a-10d7-4761-877c-46bd3df261c2-kube-api-access-vcs5g\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:32:32.006702 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f156b-c309-4b4b-8a77-270faeb79648-data-volume\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.006702 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006603 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f156b-c309-4b4b-8a77-270faeb79648-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.006702 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006636 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f156b-c309-4b4b-8a77-270faeb79648-crio-socket\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.006832 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc939343-5581-480f-8c67-37ba0d1188c6-tmp-dir\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.006832 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.006810 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc939343-5581-480f-8c67-37ba0d1188c6-config-volume\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.009577 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.009550 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f156b-c309-4b4b-8a77-270faeb79648-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.009577 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.009550 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc939343-5581-480f-8c67-37ba0d1188c6-metrics-tls\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.009764 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.009744 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0686737a-10d7-4761-877c-46bd3df261c2-cert\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:32:32.031678 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.031656 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs5g\" (UniqueName: \"kubernetes.io/projected/0686737a-10d7-4761-877c-46bd3df261c2-kube-api-access-vcs5g\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:32:32.033120 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.033100 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5l8w\" (UniqueName: \"kubernetes.io/projected/dc939343-5581-480f-8c67-37ba0d1188c6-kube-api-access-q5l8w\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.033205 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.033159 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpc4l\" (UniqueName: \"kubernetes.io/projected/db5f156b-c309-4b4b-8a77-270faeb79648-kube-api-access-cpc4l\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.050106 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.050084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:32.064225 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.064207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:32:32.092499 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.092479 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:32:32.431969 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.431925 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fqntp"] Oct 30 12:32:32.433392 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:32.433360 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0686737a_10d7_4761_877c_46bd3df261c2.slice/crio-1827a3cf2e04b66e1352f2d22df1ad4000b8986b5ab9c0422431a7d2ae9cba6b WatchSource:0}: Error finding container 1827a3cf2e04b66e1352f2d22df1ad4000b8986b5ab9c0422431a7d2ae9cba6b: Status 404 returned error can't find the container with id 1827a3cf2e04b66e1352f2d22df1ad4000b8986b5ab9c0422431a7d2ae9cba6b Oct 30 12:32:32.435005 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.434976 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8njh7"] Oct 30 12:32:32.435667 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.435645 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dxl2t"] Oct 30 12:32:32.436516 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:32.436498 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc939343_5581_480f_8c67_37ba0d1188c6.slice/crio-7d91157ee3deb2e73b1b2f6c38bc780ac83b7de2d8b24f037209c4b1fa9de8dc WatchSource:0}: Error finding container 7d91157ee3deb2e73b1b2f6c38bc780ac83b7de2d8b24f037209c4b1fa9de8dc: Status 404 returned error can't find the container with id 7d91157ee3deb2e73b1b2f6c38bc780ac83b7de2d8b24f037209c4b1fa9de8dc Oct 30 12:32:32.438777 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:32.438743 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5f156b_c309_4b4b_8a77_270faeb79648.slice/crio-016579cdb87e2e798847a0e30caac0807a8cce0bc451d5b051322e10b3925939 WatchSource:0}: Error finding container 016579cdb87e2e798847a0e30caac0807a8cce0bc451d5b051322e10b3925939: Status 404 returned error can't find the container with id 016579cdb87e2e798847a0e30caac0807a8cce0bc451d5b051322e10b3925939 Oct 30 12:32:32.925913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.925876 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:32.925913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.925905 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:32.926526 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.926132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:32.942800 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.942761 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:32:32.943247 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.943224 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:32:32.956695 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.956670 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:32:32.956885 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.956786 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zprfc\"" Oct 30 12:32:32.957082 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.957059 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wqghl\"" Oct 30 12:32:32.957189 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.957148 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sr4bw\"" Oct 30 12:32:32.957189 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:32.957181 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:32:33.239146 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:33.239064 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxl2t" event={"ID":"dc939343-5581-480f-8c67-37ba0d1188c6","Type":"ContainerStarted","Data":"7d91157ee3deb2e73b1b2f6c38bc780ac83b7de2d8b24f037209c4b1fa9de8dc"} Oct 30 12:32:33.242980 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:33.242942 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8njh7" event={"ID":"db5f156b-c309-4b4b-8a77-270faeb79648","Type":"ContainerStarted","Data":"e6f73c498b04213c11bf10c64d0c937100f211f1d8dcd66c395722fa0085d3a5"} Oct 30 12:32:33.243144 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:33.243016 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8njh7" event={"ID":"db5f156b-c309-4b4b-8a77-270faeb79648","Type":"ContainerStarted","Data":"016579cdb87e2e798847a0e30caac0807a8cce0bc451d5b051322e10b3925939"} Oct 30 12:32:33.244944 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:33.244915 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fqntp" event={"ID":"0686737a-10d7-4761-877c-46bd3df261c2","Type":"ContainerStarted","Data":"1827a3cf2e04b66e1352f2d22df1ad4000b8986b5ab9c0422431a7d2ae9cba6b"} Oct 30 12:32:34.144076 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.143959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:34.160433 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.160404 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk"] Oct 30 12:32:34.162964 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.162940 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.181018 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.180988 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-c68n6\"" Oct 30 12:32:34.182684 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.182640 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 12:32:34.196710 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.196685 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:32:34.196851 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.196799 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:32:34.196910 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.196865 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:32:34.203071 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.203032 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk"] Oct 30 12:32:34.249977 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.249937 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8njh7" event={"ID":"db5f156b-c309-4b4b-8a77-270faeb79648","Type":"ContainerStarted","Data":"be583ad9d02c4e8dbf7eff998ccdffcabd990b24e7709e46a5f122a7d0c39038"} Oct 30 12:32:34.285166 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.285139 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 12:32:34.325633 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.325599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.325796 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.325688 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.325796 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.325763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/532d8fee-b407-48ed-b9f5-77b8ad7f7630-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.325916 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.325882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhkph\" (UniqueName: \"kubernetes.io/projected/532d8fee-b407-48ed-b9f5-77b8ad7f7630-kube-api-access-nhkph\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.361143 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.361092 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:34.427438 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.426759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.427438 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.426809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/532d8fee-b407-48ed-b9f5-77b8ad7f7630-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.427438 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.426873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhkph\" (UniqueName: \"kubernetes.io/projected/532d8fee-b407-48ed-b9f5-77b8ad7f7630-kube-api-access-nhkph\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.427438 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.426928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.428010 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.427969 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/532d8fee-b407-48ed-b9f5-77b8ad7f7630-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.430493 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.430468 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.430600 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.430569 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.459092 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.459066 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhkph\" (UniqueName: \"kubernetes.io/projected/532d8fee-b407-48ed-b9f5-77b8ad7f7630-kube-api-access-nhkph\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.478777 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.478749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:32:34.805732 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:34.805576 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk"] Oct 30 12:32:34.809244 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:34.809209 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532d8fee_b407_48ed_b9f5_77b8ad7f7630.slice/crio-457ceac783c6ae2395922e13d69810b584047f05bc64e8f4314d5d57e5fd4521 WatchSource:0}: Error finding container 457ceac783c6ae2395922e13d69810b584047f05bc64e8f4314d5d57e5fd4521: Status 404 returned error can't find the container with id 457ceac783c6ae2395922e13d69810b584047f05bc64e8f4314d5d57e5fd4521 Oct 30 12:32:35.254678 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:35.254637 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxl2t" event={"ID":"dc939343-5581-480f-8c67-37ba0d1188c6","Type":"ContainerStarted","Data":"857a4dc57ca036d2836c40bef035fc6cbd96e311aafd89dcb08631d9abcef510"} Oct 30 12:32:35.255061 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:35.254707 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxl2t" event={"ID":"dc939343-5581-480f-8c67-37ba0d1188c6","Type":"ContainerStarted","Data":"034e68684ce6cab1417ab32075c88d2f8b073f786859d9249acb8a6fa0e83c6a"} Oct 30 12:32:35.255536 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:35.255517 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:35.257915 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:35.257884 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8njh7" event={"ID":"db5f156b-c309-4b4b-8a77-270faeb79648","Type":"ContainerStarted","Data":"ddb3e555cb343849d987da55b565c54552471ed9cea26b69ccbc4bb81d832f74"} Oct 30 12:32:35.259560 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:35.259528 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" event={"ID":"532d8fee-b407-48ed-b9f5-77b8ad7f7630","Type":"ContainerStarted","Data":"457ceac783c6ae2395922e13d69810b584047f05bc64e8f4314d5d57e5fd4521"} Oct 30 12:32:35.261289 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:35.261244 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fqntp" event={"ID":"0686737a-10d7-4761-877c-46bd3df261c2","Type":"ContainerStarted","Data":"9d99561167815f2791536ead38b2bb838b2a0a2075fa6f9296cfea120f4272be"} Oct 30 12:32:35.301901 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:35.301840 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dxl2t" podStartSLOduration=2.3070734010000002 podStartE2EDuration="4.301822177s" totalImagesPullingTime="1.994748776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:35.301145139 +0000 UTC m=+45.884393501" watchObservedRunningTime="2026-10-30 12:32:35.301822177 +0000 UTC m=+45.885070559" Oct 30 12:32:35.346340 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:35.346254 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fqntp" podStartSLOduration=1.6149587730000001 podStartE2EDuration="4.346234918s" totalImagesPullingTime="2.731276145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:35.345258521 +0000 UTC m=+45.928506889" watchObservedRunningTime="2026-10-30 12:32:35.346234918 +0000 UTC m=+45.929483265" Oct 30 12:32:35.391998 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:35.391920 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8njh7" podStartSLOduration=2.086168361 podStartE2EDuration="4.391905993s" totalImagesPullingTime="2.305737632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:32:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:35.391142421 +0000 UTC m=+45.974390781" watchObservedRunningTime="2026-10-30 12:32:35.391905993 +0000 UTC m=+45.975154346" Oct 30 12:32:36.270392 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:36.270358 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" event={"ID":"532d8fee-b407-48ed-b9f5-77b8ad7f7630","Type":"ContainerStarted","Data":"cfe72e1008ed86d48aec6b880d74a7e50f5b96d11fff78ec78feb5b8ecae520a"} Oct 30 12:32:36.270872 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:36.270428 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" event={"ID":"532d8fee-b407-48ed-b9f5-77b8ad7f7630","Type":"ContainerStarted","Data":"2ef3cf72a6e8c7e981f45676952e04adf272ecb77d2a389b04cc67a610be5775"} Oct 30 12:32:36.314774 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:36.314726 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" podStartSLOduration=1.283206019 podStartE2EDuration="2.314713011s" totalImagesPullingTime="1.031506992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:36.314567269 +0000 UTC m=+46.897815616" watchObservedRunningTime="2026-10-30 12:32:36.314713011 +0000 UTC m=+46.897961358" Oct 30 12:32:36.812679 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:36.812640 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:37.205883 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:37.205848 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:37.412313 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:37.412282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:38.658132 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:38.658079 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:38.901061 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:38.901012 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:38.927812 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:38.927748 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q2vhb"] Oct 30 12:32:38.929778 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:38.929763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:38.998609 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:38.998574 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:32:38.998800 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:38.998679 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f8qlc\"" Oct 30 12:32:38.998800 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:38.998726 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:32:38.998909 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:38.998826 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:32:39.066241 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.066202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-wtmp\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.066241 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.066239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdpzv\" (UniqueName: \"kubernetes.io/projected/5b73fd1c-6f57-4172-b98d-c8b8330efd83-kube-api-access-sdpzv\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.066444 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.066301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-sys\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.066444 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.066339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-textfile\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.066444 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.066373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-metrics-client-ca\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.066537 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.066464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-tls\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.066537 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.066483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.066595 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.066542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-root\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.066595 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.066567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.106232 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.106204 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:39.167891 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.167858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-wtmp\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.167891 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.167891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdpzv\" (UniqueName: \"kubernetes.io/projected/5b73fd1c-6f57-4172-b98d-c8b8330efd83-kube-api-access-sdpzv\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168121 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.167928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-sys\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168121 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.167950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-textfile\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168121 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.167972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-metrics-client-ca\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168121 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-tls\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168121 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168008 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-sys\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168121 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168121 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168054 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-wtmp\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168121 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-root\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168486 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168134 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168486 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168169 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-root\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168486 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168405 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-textfile\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168705 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168680 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-metrics-client-ca\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.168705 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.168691 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.170310 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.170291 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.170415 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.170396 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-tls\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.193388 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.193320 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdpzv\" (UniqueName: \"kubernetes.io/projected/5b73fd1c-6f57-4172-b98d-c8b8330efd83-kube-api-access-sdpzv\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.242155 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.242108 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:32:39.252675 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:39.252643 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b73fd1c_6f57_4172_b98d_c8b8330efd83.slice/crio-d670e49694a2f90b60b06edb9289e84cdde05c46fcfc100f8f79d4648dee2fc3 WatchSource:0}: Error finding container d670e49694a2f90b60b06edb9289e84cdde05c46fcfc100f8f79d4648dee2fc3: Status 404 returned error can't find the container with id d670e49694a2f90b60b06edb9289e84cdde05c46fcfc100f8f79d4648dee2fc3 Oct 30 12:32:39.283964 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.283929 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2vhb" event={"ID":"5b73fd1c-6f57-4172-b98d-c8b8330efd83","Type":"ContainerStarted","Data":"d670e49694a2f90b60b06edb9289e84cdde05c46fcfc100f8f79d4648dee2fc3"} Oct 30 12:32:39.324513 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.324477 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:39.537290 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:39.537191 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:40.037260 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.037222 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:40.255676 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.255347 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:40.271074 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.271050 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:32:40.273660 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.273643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.277877 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.277854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.277968 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.277887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.277968 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.277929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.277968 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.277949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.278084 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.278066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.278114 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.278102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.278147 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.278119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.278227 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.278208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddf98\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.278287 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.278256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.278326 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.278294 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.278326 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.278311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.278383 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.278327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.278418 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.278406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.288746 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.288698 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b73fd1c-6f57-4172-b98d-c8b8330efd83" containerID="52105d3443d20f4855ad1b3f5281c7acfc59c76b68bcbebbd51d6f0145d18a2e" exitCode=0 Oct 30 12:32:40.288814 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.288752 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2vhb" event={"ID":"5b73fd1c-6f57-4172-b98d-c8b8330efd83","Type":"ContainerDied","Data":"52105d3443d20f4855ad1b3f5281c7acfc59c76b68bcbebbd51d6f0145d18a2e"} Oct 30 12:32:40.297963 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.297943 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 12:32:40.298091 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.298074 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 12:32:40.314366 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.314340 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:32:40.327744 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.327723 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 12:32:40.378699 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.378673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.378796 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.378703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.378899 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.378879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.378955 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.378930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379050 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379114 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddf98\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379166 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379166 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379251 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379251 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379378 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379378 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379321 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379378 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379526 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379662 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379634 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.379970 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.379950 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.381117 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.381093 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.381482 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.381466 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.381533 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.381505 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.394124 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.394104 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 12:32:40.394207 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.394126 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 12:32:40.394207 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.394126 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 12:32:40.394332 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.394300 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 12:32:40.394375 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.394362 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pqqdj\"" Oct 30 12:32:40.394406 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.394399 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 12:32:40.394812 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.394801 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 12:32:40.401581 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.401557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.401747 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.401581 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.401747 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.401563 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.401902 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.401886 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.402019 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.402001 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.402053 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.402028 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.406846 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.406827 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddf98\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.471698 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.471665 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:40.645981 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.645950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:32:40.967447 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:40.967370 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:32:40.978811 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:40.978788 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea59390_80c3_432c_bda5_88af53de9263.slice/crio-6d965288c14cb8d369763f0f7b69aa1295310a7bcc9177dd32191a483e8987f9 WatchSource:0}: Error finding container 6d965288c14cb8d369763f0f7b69aa1295310a7bcc9177dd32191a483e8987f9: Status 404 returned error can't find the container with id 6d965288c14cb8d369763f0f7b69aa1295310a7bcc9177dd32191a483e8987f9 Oct 30 12:32:41.065355 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.065318 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55b7b78d57-rlt5q"] Oct 30 12:32:41.068727 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.068711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.084382 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.084356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.084503 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.084385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbafdcd8-268f-4128-b17e-10d3d92e2227-metrics-client-ca\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.084503 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.084403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7rm\" (UniqueName: \"kubernetes.io/projected/dbafdcd8-268f-4128-b17e-10d3d92e2227-kube-api-access-8r7rm\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.084503 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.084425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.084640 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.084504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-grpc-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.084640 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.084631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbafdcd8-268f-4128-b17e-10d3d92e2227-tmp\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.084706 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.084653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.084706 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.084674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.084706 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.084692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.086584 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.086557 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 12:32:41.086584 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.086569 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 12:32:41.086584 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.086565 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 12:32:41.086783 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.086564 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 12:32:41.086783 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.086564 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-47x29\"" Oct 30 12:32:41.086783 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.086556 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-57lkoqn2g03t5\"" Oct 30 12:32:41.086783 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.086636 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 12:32:41.098105 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.098077 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55b7b78d57-rlt5q"] Oct 30 12:32:41.186050 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186238 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbafdcd8-268f-4128-b17e-10d3d92e2227-metrics-client-ca\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186238 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7rm\" (UniqueName: \"kubernetes.io/projected/dbafdcd8-268f-4128-b17e-10d3d92e2227-kube-api-access-8r7rm\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186238 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186238 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-grpc-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186238 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbafdcd8-268f-4128-b17e-10d3d92e2227-tmp\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186506 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186506 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186297 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186506 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186656 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186622 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbafdcd8-268f-4128-b17e-10d3d92e2227-tmp\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.186896 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.186871 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbafdcd8-268f-4128-b17e-10d3d92e2227-metrics-client-ca\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.188775 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.188745 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.189040 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.189018 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.189388 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.189364 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.189507 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.189485 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.189581 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.189561 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.189581 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.189573 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-grpc-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.211471 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.211441 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7rm\" (UniqueName: \"kubernetes.io/projected/dbafdcd8-268f-4128-b17e-10d3d92e2227-kube-api-access-8r7rm\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.293645 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.293562 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerStarted","Data":"6d965288c14cb8d369763f0f7b69aa1295310a7bcc9177dd32191a483e8987f9"} Oct 30 12:32:41.295903 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.295871 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2vhb" event={"ID":"5b73fd1c-6f57-4172-b98d-c8b8330efd83","Type":"ContainerStarted","Data":"65f4b322ec168ee2b642dc69e04eacfee89cb8bd4bf91495aba8141341ab0c1a"} Oct 30 12:32:41.296080 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.295918 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2vhb" event={"ID":"5b73fd1c-6f57-4172-b98d-c8b8330efd83","Type":"ContainerStarted","Data":"36914454a783514d9ad5afc76f51e9a7ad1f5f101d52826ec3369ed1af026f96"} Oct 30 12:32:41.343388 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.343342 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q2vhb" podStartSLOduration=2.389737343 podStartE2EDuration="3.343324935s" totalImagesPullingTime="885.689441ms" totalInitContainerRuntime="67.898151ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:41.342230003 +0000 UTC m=+51.925478349" watchObservedRunningTime="2026-10-30 12:32:41.343324935 +0000 UTC m=+51.926573283" Oct 30 12:32:41.384027 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.383984 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:41.706195 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:41.706161 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55b7b78d57-rlt5q"] Oct 30 12:32:41.707614 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:41.707587 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbafdcd8_268f_4128_b17e_10d3d92e2227.slice/crio-8475c423ff1416280b35d5a6d2685a3fcf4c351b98060e4cf12ebdc98196b3bc WatchSource:0}: Error finding container 8475c423ff1416280b35d5a6d2685a3fcf4c351b98060e4cf12ebdc98196b3bc: Status 404 returned error can't find the container with id 8475c423ff1416280b35d5a6d2685a3fcf4c351b98060e4cf12ebdc98196b3bc Oct 30 12:32:42.207651 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:42.207617 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:42.301633 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:42.301593 2575 generic.go:356] "Generic (PLEG): container finished" podID="4ea59390-80c3-432c-bda5-88af53de9263" containerID="4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024" exitCode=0 Oct 30 12:32:42.301816 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:42.301705 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerDied","Data":"4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024"} Oct 30 12:32:42.303043 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:42.303010 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" event={"ID":"dbafdcd8-268f-4128-b17e-10d3d92e2227","Type":"ContainerStarted","Data":"8475c423ff1416280b35d5a6d2685a3fcf4c351b98060e4cf12ebdc98196b3bc"} Oct 30 12:32:42.450915 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:42.450852 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:43.649463 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.649442 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs"] Oct 30 12:32:43.651367 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.651349 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:43.668623 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.668596 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 12:32:43.674397 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.674359 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-l82fp\"" Oct 30 12:32:43.676184 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.676160 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs"] Oct 30 12:32:43.708724 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.708697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/475cbe1a-bcd8-4575-9c39-3c3394f062e4-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:43.708876 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.708769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/475cbe1a-bcd8-4575-9c39-3c3394f062e4-kube-api-access-j8jr9\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:43.739074 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.739046 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:43.809744 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.809698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/475cbe1a-bcd8-4575-9c39-3c3394f062e4-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:43.809834 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.809760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/475cbe1a-bcd8-4575-9c39-3c3394f062e4-kube-api-access-j8jr9\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:43.812843 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.812817 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/475cbe1a-bcd8-4575-9c39-3c3394f062e4-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:43.838615 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.838529 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/475cbe1a-bcd8-4575-9c39-3c3394f062e4-kube-api-access-j8jr9\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:43.956433 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.956326 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:43.972840 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:43.972807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:44.208330 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.208281 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:32:44.278324 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:44.278300 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475cbe1a_bcd8_4575_9c39_3c3394f062e4.slice/crio-9835506cea0d204c5020c73b46fc2b605165090ce6b34c2b0ddd23260117a198 WatchSource:0}: Error finding container 9835506cea0d204c5020c73b46fc2b605165090ce6b34c2b0ddd23260117a198: Status 404 returned error can't find the container with id 9835506cea0d204c5020c73b46fc2b605165090ce6b34c2b0ddd23260117a198 Oct 30 12:32:44.279364 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.279343 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs"] Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.319642 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerStarted","Data":"c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39"} Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.319736 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerStarted","Data":"7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4"} Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.319766 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerStarted","Data":"400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4"} Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.319793 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerStarted","Data":"498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846"} Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.319818 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerStarted","Data":"63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac"} Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.322902 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" event={"ID":"dbafdcd8-268f-4128-b17e-10d3d92e2227","Type":"ContainerStarted","Data":"e30079001fb384c28ee3111e884236477efba71d4d10dacb6c82e522530ab023"} Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.322979 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" event={"ID":"dbafdcd8-268f-4128-b17e-10d3d92e2227","Type":"ContainerStarted","Data":"f8e4b36a2719be73d0875bfbe7e99a9879e7959d61cc2424d678476916cacbc8"} Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.323018 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" event={"ID":"dbafdcd8-268f-4128-b17e-10d3d92e2227","Type":"ContainerStarted","Data":"ccd56fbcb9ee514189cbe2c1ef2ee0c8d7591aa3ebf37457fc2dfa616887663e"} Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.323057 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" event={"ID":"dbafdcd8-268f-4128-b17e-10d3d92e2227","Type":"ContainerStarted","Data":"a7e109d414c1bb1b0e6c4618f2e2f233b0e6183eb638a8dea358cf5c85097485"} Oct 30 12:32:44.323459 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.323095 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" event={"ID":"dbafdcd8-268f-4128-b17e-10d3d92e2227","Type":"ContainerStarted","Data":"aab83e3beaf144948f5d253a23946f5a2a9836bbd270746fc1bf328b408026bd"} Oct 30 12:32:44.324425 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:44.324398 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" event={"ID":"475cbe1a-bcd8-4575-9c39-3c3394f062e4","Type":"ContainerStarted","Data":"9835506cea0d204c5020c73b46fc2b605165090ce6b34c2b0ddd23260117a198"} Oct 30 12:32:45.335136 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:45.335097 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" event={"ID":"dbafdcd8-268f-4128-b17e-10d3d92e2227","Type":"ContainerStarted","Data":"df847034ce72e45b91fb1f41fab48c9dc1b8a2cde8b870ff96f941d66e24962c"} Oct 30 12:32:45.335625 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:45.335223 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:45.340457 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:45.340419 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerStarted","Data":"1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48"} Oct 30 12:32:45.427958 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:45.427896 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" podStartSLOduration=2.138211655 podStartE2EDuration="4.427883052s" totalImagesPullingTime="2.289671397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:45.413053958 +0000 UTC m=+55.996302330" watchObservedRunningTime="2026-10-30 12:32:45.427883052 +0000 UTC m=+56.011131428" Oct 30 12:32:45.470582 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:45.470536 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.8508715860000002 podStartE2EDuration="5.470522101s" totalImagesPullingTime="2.604414361s" totalInitContainerRuntime="15.236154ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:45.470204748 +0000 UTC m=+56.053453115" watchObservedRunningTime="2026-10-30 12:32:45.470522101 +0000 UTC m=+56.053770461" Oct 30 12:32:45.907794 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:45.907759 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:46.277989 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:46.277915 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dxl2t" Oct 30 12:32:46.348219 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:46.348176 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" event={"ID":"475cbe1a-bcd8-4575-9c39-3c3394f062e4","Type":"ContainerStarted","Data":"6640377ab99f39efee31c68c9af3f250673a37f8c7d31fea02999a638f435d96"} Oct 30 12:32:46.348674 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:46.348296 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:46.357277 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:46.357234 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:32:46.393383 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:46.393336 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" podStartSLOduration=2.221214278 podStartE2EDuration="3.393322912s" totalImagesPullingTime="1.172108634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:46.392603808 +0000 UTC m=+56.975852154" watchObservedRunningTime="2026-10-30 12:32:46.393322912 +0000 UTC m=+56.976571258" Oct 30 12:32:50.685739 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:50.685704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:50.890176 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:50.890142 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:51.356078 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:51.356051 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:32:52.210119 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:52.210083 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:52.416767 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:52.416736 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:55.693001 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.692959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:55.713592 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.713560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:55.713728 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.713609 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:55.729236 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.729207 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:32:55.736512 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.736485 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:55.743180 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.743158 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:32:55.746174 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.746155 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:55.796174 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.796138 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zprfc\"" Oct 30 12:32:55.801290 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.801261 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:32:55.807577 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.807557 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sr4bw\"" Oct 30 12:32:55.812701 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.812678 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:32:55.814895 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.814842 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:55.830772 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.830747 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:32:55.851667 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.851637 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:32:55.859846 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.859722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:55.904736 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:55.904703 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:56.070089 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.069998 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wqghl\"" Oct 30 12:32:56.075082 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.075051 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:32:56.110920 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.110891 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2j4ks"] Oct 30 12:32:56.112947 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:56.112910 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102a0155_f391_4c1e_98d7_eea36a51615e.slice/crio-ddc1fb8a75fd76c11f1a215eda5e5ae2d9b47e2a6f3c37093187b03b32e133a2 WatchSource:0}: Error finding container ddc1fb8a75fd76c11f1a215eda5e5ae2d9b47e2a6f3c37093187b03b32e133a2: Status 404 returned error can't find the container with id ddc1fb8a75fd76c11f1a215eda5e5ae2d9b47e2a6f3c37093187b03b32e133a2 Oct 30 12:32:56.119674 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.119626 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8fl8b"] Oct 30 12:32:56.122030 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:56.122004 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc1b4c7_ef74_4105_b613_2f53486e7aad.slice/crio-a454ceec44bbe20b1030089ab27e07caf34ec226cb748bd76f202ca7d5bb554c WatchSource:0}: Error finding container a454ceec44bbe20b1030089ab27e07caf34ec226cb748bd76f202ca7d5bb554c: Status 404 returned error can't find the container with id a454ceec44bbe20b1030089ab27e07caf34ec226cb748bd76f202ca7d5bb554c Oct 30 12:32:56.148759 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.148723 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:56.355558 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.355468 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:56.367203 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.367180 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjjj6"] Oct 30 12:32:56.367993 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:56.367968 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88627b5e_75be_4029_ba55_51759aedc92d.slice/crio-c97a632f2a75fa873d60632942041c9abb69af13a3cc44cc510b70c59783a85d WatchSource:0}: Error finding container c97a632f2a75fa873d60632942041c9abb69af13a3cc44cc510b70c59783a85d: Status 404 returned error can't find the container with id c97a632f2a75fa873d60632942041c9abb69af13a3cc44cc510b70c59783a85d Oct 30 12:32:56.381627 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.381596 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjjj6" event={"ID":"88627b5e-75be-4029-ba55-51759aedc92d","Type":"ContainerStarted","Data":"c97a632f2a75fa873d60632942041c9abb69af13a3cc44cc510b70c59783a85d"} Oct 30 12:32:56.382985 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.382888 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2j4ks" event={"ID":"102a0155-f391-4c1e-98d7-eea36a51615e","Type":"ContainerStarted","Data":"ddc1fb8a75fd76c11f1a215eda5e5ae2d9b47e2a6f3c37093187b03b32e133a2"} Oct 30 12:32:56.384139 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.384109 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8fl8b" event={"ID":"ecc1b4c7-ef74-4105-b613-2f53486e7aad","Type":"ContainerStarted","Data":"a454ceec44bbe20b1030089ab27e07caf34ec226cb748bd76f202ca7d5bb554c"} Oct 30 12:32:56.563102 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.563064 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:56.862622 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.862584 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-s2gdv"] Oct 30 12:32:56.865093 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.865067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:32:56.886158 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.886131 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-5p94n\"" Oct 30 12:32:56.888920 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.888893 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 12:32:56.902277 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.902232 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-s2gdv"] Oct 30 12:32:56.902463 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.902436 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 12:32:56.925445 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.925410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-tmp\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:32:56.925594 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:56.925511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2k6\" (UniqueName: \"kubernetes.io/projected/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-kube-api-access-hr2k6\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:32:57.026868 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.026836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-tmp\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:32:57.027034 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.026905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2k6\" (UniqueName: \"kubernetes.io/projected/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-kube-api-access-hr2k6\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:32:57.027327 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.027292 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-tmp\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:32:57.055490 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.055463 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2k6\" (UniqueName: \"kubernetes.io/projected/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-kube-api-access-hr2k6\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:32:57.180038 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.180005 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:32:57.390166 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.390132 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2j4ks" event={"ID":"102a0155-f391-4c1e-98d7-eea36a51615e","Type":"ContainerStarted","Data":"4509be94425362b1a9a3887c1354609252bb7e4db99af2b0531d1811eab0f133"} Oct 30 12:32:57.390358 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.390184 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2j4ks" event={"ID":"102a0155-f391-4c1e-98d7-eea36a51615e","Type":"ContainerStarted","Data":"63f11331ef8fd05fbe071f8ea36404f342dc8c9894549e00bf4354fc236eb6e0"} Oct 30 12:32:57.434668 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.434565 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2j4ks" podStartSLOduration=66.582808265 podStartE2EDuration="1m7.434546624s" totalImagesPullingTime="851.738359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:57.433375869 +0000 UTC m=+68.016624216" watchObservedRunningTime="2026-10-30 12:32:57.434546624 +0000 UTC m=+68.017794971" Oct 30 12:32:57.483979 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.483940 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:57.495594 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.495300 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-s2gdv"] Oct 30 12:32:57.498375 ip-10-0-134-193 kubenswrapper[2575]: W1030 12:32:57.498338 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983c8f5c_68cf_42a4_b931_4ceb2b7e1e77.slice/crio-649e56f7044427264966013bb115f28e137d727d9b61aecbe8dae8c1f405bfd1 WatchSource:0}: Error finding container 649e56f7044427264966013bb115f28e137d727d9b61aecbe8dae8c1f405bfd1: Status 404 returned error can't find the container with id 649e56f7044427264966013bb115f28e137d727d9b61aecbe8dae8c1f405bfd1 Oct 30 12:32:57.690859 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:57.690775 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:58.401542 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:32:58.401435 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-s2gdv" event={"ID":"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77","Type":"ContainerStarted","Data":"649e56f7044427264966013bb115f28e137d727d9b61aecbe8dae8c1f405bfd1"} Oct 30 12:33:00.412837 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:00.412799 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjjj6" event={"ID":"88627b5e-75be-4029-ba55-51759aedc92d","Type":"ContainerStarted","Data":"16ec153bc9e1144ddade344af54237851f79a7f87baa974edcf3b237cd9559eb"} Oct 30 12:33:00.413239 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:00.413052 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:33:00.457110 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:00.457050 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kjjj6" podStartSLOduration=66.830669667 podStartE2EDuration="1m10.457033101s" totalImagesPullingTime="3.626363434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:00.455827433 +0000 UTC m=+71.039075780" watchObservedRunningTime="2026-10-30 12:33:00.457033101 +0000 UTC m=+71.040281448" Oct 30 12:33:01.154979 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.154939 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:01.360664 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.360591 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:01.594793 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.594762 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8b6c8b7bd-8d84r"] Oct 30 12:33:01.597360 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.597324 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.615730 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.615697 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 12:33:01.615730 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.615711 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 12:33:01.615730 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.615710 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 12:33:01.631344 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.631308 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b6c8b7bd-8d84r"] Oct 30 12:33:01.671832 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.671785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.672019 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.671839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.672019 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.671930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.672019 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.671963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.672212 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.672097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.672212 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.672147 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.672379 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.672261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.702913 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.702830 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 12:33:01.703093 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.702942 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 12:33:01.703093 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.702997 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2gf2w\"" Oct 30 12:33:01.773052 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.773223 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.773223 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.773223 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.773223 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.773223 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.773513 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.773862 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773836 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.774004 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773896 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.774004 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.773958 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.774186 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.774125 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.776040 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.776015 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.776402 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.776376 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.802560 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.802532 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:01.810811 ip-10-0-134-193 kubenswrapper[2575]: I1030 12:33:01.810728 2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:33:01.810883 ip-10-0-134-193 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 12:33:01.822799 ip-10-0-134-193 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 12:33:01.823036 ip-10-0-134-193 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 12:33:01.823090 ip-10-0-134-193 systemd[1]: kubelet.service: Consumed 4.271s CPU time. Oct 30 12:33:01.839823 ip-10-0-134-193 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:33:01.904515 ip-10-0-134-193 kubenswrapper[5515]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:33:01.904515 ip-10-0-134-193 kubenswrapper[5515]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:33:01.904515 ip-10-0-134-193 kubenswrapper[5515]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:33:01.904515 ip-10-0-134-193 kubenswrapper[5515]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:33:01.907862 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907763 5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:33:01.907862 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907857 5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:33:01.907862 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907861 5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:33:01.907862 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907865 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:33:01.907862 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907869 5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:33:01.907862 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907872 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907876 5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907879 5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907882 5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907885 5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907888 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907890 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907894 5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907897 5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907900 5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907903 5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907905 5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907908 5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907911 5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907914 5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907917 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907920 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907922 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907925 5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907928 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:33:01.908113 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907931 5515 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907934 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907936 5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907939 5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907942 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907944 5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907947 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907951 5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907954 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907956 5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907959 5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907962 5515 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907964 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907967 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907970 5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907973 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907975 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907978 5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907980 5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907983 5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:33:01.908661 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907985 5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907988 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907991 5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907993 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907996 5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.907998 5515 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908001 5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908004 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908006 5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908009 5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908012 5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908016 5515 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908018 5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908021 5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908024 5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908026 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908029 5515 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908031 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908034 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908036 5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:33:01.909152 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908039 5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908041 5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908044 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908046 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908049 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908051 5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908054 5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908057 5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908059 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908062 5515 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908073 5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908077 5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908080 5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908083 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908086 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908088 5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908091 5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908095 5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908099 5515 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:33:01.909666 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908101 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908107 5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908110 5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908114 5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908118 5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908120 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908123 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908127 5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908130 5515 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908133 5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908135 5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908138 5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908141 5515 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908144 5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908147 5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908151 5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908155 5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908159 5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908162 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:33:01.910174 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908165 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908168 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908171 5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908967 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908975 5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908979 5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908982 5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908984 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908987 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908990 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908993 5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908996 5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.908999 5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909002 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909005 5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909008 5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909010 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909014 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909017 5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:33:01.910682 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909019 5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909022 5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909025 5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909027 5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909030 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909032 5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909035 5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909038 5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909040 5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909043 5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909045 5515 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909048 5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909051 5515 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909053 5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909056 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909059 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909062 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909064 5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909067 5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909070 5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:33:01.911160 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909073 5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909076 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909078 5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909081 5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909084 5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909086 5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909091 5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909095 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909098 5515 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909101 5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909104 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909107 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909109 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909112 5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909114 5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909117 5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909119 5515 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909122 5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909126 5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:33:01.911689 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909129 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909132 5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909135 5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909138 5515 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909141 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909143 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909146 5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909148 5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909151 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909155 5515 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909158 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909161 5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909164 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909167 5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909169 5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909172 5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909176 5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909179 5515 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909182 5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909185 5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:33:01.912159 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909187 5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909190 5515 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909193 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909196 5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909198 5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909201 5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909204 5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909207 5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909209 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909212 5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909214 5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909217 5515 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909219 5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909222 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909224 5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909227 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909229 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909232 5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909235 5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909237 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:33:01.912720 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909240 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909243 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909246 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909249 5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909252 5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909255 5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909257 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909282 5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909286 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909289 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909291 5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909852 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909857 5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909862 5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909867 5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909871 5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909874 5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909878 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909881 5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909883 5515 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:33:01.913309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909886 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909889 5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909891 5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909894 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909897 5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909900 5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909903 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909905 5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909908 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909910 5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909913 5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909916 5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909919 5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909921 5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909924 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909926 5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909929 5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909931 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909934 5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909938 5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:33:01.913796 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909941 5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909944 5515 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909946 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909949 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909951 5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909954 5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909959 5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909963 5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909966 5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909968 5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909971 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909973 5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909977 5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909979 5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909982 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909984 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909987 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909990 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909992 5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:33:01.914326 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909995 5515 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909997 5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.909999 5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910002 5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910005 5515 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910007 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910010 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910012 5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910015 5515 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910017 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910020 5515 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910023 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910026 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910030 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910033 5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910035 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910038 5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910041 5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910043 5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910046 5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:33:01.914784 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910048 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910051 5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910054 5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910056 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910059 5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910061 5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910064 5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910066 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910069 5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910072 5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910074 5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910077 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910079 5515 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910083 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910086 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910089 5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910091 5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910094 5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910097 5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:33:01.915283 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910099 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910102 5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910105 5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910108 5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910111 5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910118 5515 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910121 5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910124 5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910126 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910129 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910132 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910134 5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910137 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910140 5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910143 5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910145 5515 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910148 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910151 5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910153 5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910245 5515 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:33:01.915775 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910257 5515 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910279 5515 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910287 5515 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910299 5515 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910303 5515 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910308 5515 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910313 5515 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910317 5515 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910320 5515 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910325 5515 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910328 5515 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910331 5515 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910334 5515 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910338 5515 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910341 5515 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910344 5515 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910347 5515 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910353 5515 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910356 5515 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910360 5515 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910363 5515 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910367 5515 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910371 5515 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:33:01.916291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910374 5515 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910377 5515 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910380 5515 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910384 5515 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910388 5515 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910391 5515 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910394 5515 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910397 5515 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910402 5515 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910405 5515 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910407 5515 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910410 5515 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910414 5515 flags.go:64] FLAG: --enable-server="true" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910416 5515 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910421 5515 flags.go:64] FLAG: --event-burst="100" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910427 5515 flags.go:64] FLAG: --event-qps="50" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910430 5515 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910434 5515 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910437 5515 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910442 5515 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910445 5515 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910449 5515 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910452 5515 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910455 5515 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910458 5515 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:33:01.916844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910461 5515 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910464 5515 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910467 5515 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910470 5515 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910474 5515 flags.go:64] FLAG: --feature-gates="" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910478 5515 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910481 5515 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910485 5515 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910488 5515 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910491 5515 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910494 5515 flags.go:64] FLAG: --help="false" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910497 5515 flags.go:64] FLAG: --hostname-override="ip-10-0-134-193.ec2.internal" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910501 5515 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910504 5515 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910507 5515 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910511 5515 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910514 5515 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910517 5515 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910520 5515 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910523 5515 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910526 5515 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910529 5515 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910534 5515 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910537 5515 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:33:01.917476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910540 5515 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910543 5515 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910546 5515 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910550 5515 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910553 5515 flags.go:64] FLAG: --lock-file="" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910556 5515 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910559 5515 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910562 5515 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910568 5515 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910571 5515 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910574 5515 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910578 5515 flags.go:64] FLAG: --logging-format="text" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910581 5515 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910585 5515 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910588 5515 flags.go:64] FLAG: --manifest-url="" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910591 5515 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910596 5515 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910599 5515 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910603 5515 flags.go:64] FLAG: --max-pods="110" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910606 5515 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910610 5515 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910613 5515 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910616 5515 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910620 5515 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910623 5515 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:33:01.918050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910626 5515 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910637 5515 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910641 5515 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910644 5515 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910647 5515 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910652 5515 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910655 5515 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910658 5515 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910661 5515 flags.go:64] FLAG: --port="10250" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910664 5515 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910667 5515 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eda2151ec4cb49c9" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910670 5515 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910673 5515 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910677 5515 flags.go:64] FLAG: --register-node="true" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910680 5515 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910684 5515 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910687 5515 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910691 5515 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910694 5515 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910698 5515 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910701 5515 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910705 5515 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910707 5515 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910710 5515 flags.go:64] FLAG: --runonce="false" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910713 5515 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910717 5515 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910720 5515 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:33:01.918655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910723 5515 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910726 5515 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910730 5515 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910733 5515 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910736 5515 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910739 5515 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910743 5515 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910747 5515 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910750 5515 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910753 5515 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910756 5515 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910759 5515 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910764 5515 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910767 5515 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910770 5515 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910775 5515 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910778 5515 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910781 5515 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910784 5515 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910787 5515 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910791 5515 flags.go:64] FLAG: --v="2" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910795 5515 flags.go:64] FLAG: --version="false" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910800 5515 flags.go:64] FLAG: --vmodule="" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910804 5515 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.910807 5515 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:33:01.919291 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910944 5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910947 5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910950 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910953 5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910955 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910958 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910960 5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910963 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910966 5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910968 5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910972 5515 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910974 5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910977 5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910982 5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910985 5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910988 5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910990 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910993 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910996 5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:33:01.919903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.910998 5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911001 5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911004 5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911007 5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911009 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911011 5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911015 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911017 5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911020 5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911023 5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911026 5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911028 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911031 5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911034 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911037 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911039 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911042 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911044 5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911047 5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:33:01.920411 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911049 5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911052 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911054 5515 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911057 5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911060 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911064 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911066 5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911070 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911073 5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911077 5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911081 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911084 5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911087 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911091 5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911094 5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911097 5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911100 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911103 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911105 5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:33:01.920859 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911108 5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911111 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911113 5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911116 5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911119 5515 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911122 5515 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911124 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911127 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911129 5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911132 5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911134 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911137 5515 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911140 5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911142 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911145 5515 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911147 5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911150 5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911153 5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911155 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911159 5515 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:33:01.921343 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911163 5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911166 5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911169 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911172 5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911174 5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911177 5515 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911180 5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911182 5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911185 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911190 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911193 5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911196 5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911198 5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911201 5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911204 5515 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911206 5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911209 5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911212 5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911215 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911217 5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:33:01.922119 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911220 5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911222 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911225 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911228 5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911230 5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911233 5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911236 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911238 5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911242 5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.911289 5515 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911341 5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911347 5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911354 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911357 5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911360 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:33:01.922855 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911363 5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911366 5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911369 5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911372 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911375 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911377 5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911380 5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911384 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911387 5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911390 5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911392 5515 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911395 5515 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911398 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911400 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911403 5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911406 5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911408 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911411 5515 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911414 5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911416 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:33:01.923635 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911419 5515 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911421 5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911424 5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911427 5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911429 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911432 5515 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911435 5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911437 5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911440 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911444 5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911453 5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911456 5515 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911459 5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911461 5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911465 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911467 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911470 5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911473 5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911475 5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911479 5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:33:01.924308 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911482 5515 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911485 5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911487 5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911490 5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911492 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911495 5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911497 5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911500 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911503 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911505 5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911508 5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911510 5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911513 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911515 5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911519 5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911522 5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911525 5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911528 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911530 5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:33:01.925028 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911533 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911536 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911540 5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911542 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911550 5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911553 5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911556 5515 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911558 5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911561 5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911565 5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911567 5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911570 5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911574 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911577 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911580 5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911582 5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911585 5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911588 5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911590 5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911594 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:33:01.925902 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911596 5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911599 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911601 5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911605 5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911607 5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911610 5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911612 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911615 5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911618 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911621 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911623 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911626 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911628 5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911631 5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911635 5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911638 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911640 5515 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911644 5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911648 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:33:01.926581 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911651 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911653 5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.911656 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.916095 5515 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.916111 5515 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.916131 5515 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.916136 5515 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916216 5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916221 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916225 5515 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916228 5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916231 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916234 5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916237 5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916240 5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916243 5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916245 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916248 5515 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916251 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:33:01.927140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916253 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916256 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916259 5515 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916262 5515 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916278 5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916281 5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916284 5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916287 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916290 5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916293 5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916296 5515 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916305 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916309 5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916313 5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916318 5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916321 5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916324 5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916328 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916331 5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916334 5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:33:01.927665 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916336 5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916339 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916342 5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916345 5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916347 5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916350 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916353 5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916355 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916358 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916361 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916363 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916366 5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916368 5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916371 5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916373 5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916376 5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916379 5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916381 5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916384 5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:33:01.928228 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916386 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916389 5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916392 5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916395 5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916397 5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916406 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916409 5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916412 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916415 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916418 5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916421 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916423 5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916426 5515 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916428 5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916431 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916434 5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916436 5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916439 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916442 5515 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916444 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:33:01.928860 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916447 5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916449 5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916452 5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916454 5515 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916457 5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916460 5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916463 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916466 5515 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916468 5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916471 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916473 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916476 5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916479 5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916481 5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916484 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916486 5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916489 5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916491 5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916500 5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916503 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:33:01.929364 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916506 5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916509 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916511 5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916514 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916517 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916519 5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916522 5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916525 5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916527 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916530 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916532 5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916536 5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916540 5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916543 5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916546 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:33:01.929837 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.916574 5515 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916764 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916770 5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916773 5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916776 5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916779 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916782 5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916785 5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916788 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916790 5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916793 5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916796 5515 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916798 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916801 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916804 5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916806 5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916815 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916817 5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916820 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916823 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:33:01.930191 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916825 5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916828 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916830 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916833 5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916835 5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916838 5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916840 5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916843 5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916846 5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916849 5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916852 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916855 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916857 5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916861 5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916864 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916866 5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916869 5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916871 5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916874 5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:33:01.930680 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916877 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916880 5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916882 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916885 5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916887 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916890 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916893 5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916896 5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916898 5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916902 5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916913 5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916917 5515 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916921 5515 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916925 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916929 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916933 5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916937 5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916940 5515 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916943 5515 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916946 5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:33:01.931140 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916949 5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916951 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916954 5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916956 5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916960 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916962 5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916965 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916968 5515 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916970 5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916973 5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916976 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916978 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916982 5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916986 5515 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916989 5515 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916992 5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916995 5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.916998 5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917001 5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:33:01.931645 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917004 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917007 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917010 5515 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917012 5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917021 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917024 5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917027 5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917031 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917034 5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917036 5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917039 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917042 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917045 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917047 5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917050 5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917052 5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917055 5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917057 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917060 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:33:01.932103 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917062 5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917065 5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917068 5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917070 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917073 5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917076 5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917078 5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917081 5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917083 5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917086 5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.917111 5515 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917157 5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917161 5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917165 5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917173 5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917177 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:33:01.932567 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917179 5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917182 5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917190 5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917193 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917196 5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917198 5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917201 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917204 5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917207 5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917209 5515 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917212 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917214 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917217 5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917219 5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917222 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917225 5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917227 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917230 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917232 5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917235 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:33:01.932956 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917237 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917240 5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917246 5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917248 5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917251 5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917254 5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917256 5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917259 5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917262 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917284 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917288 5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917291 5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917293 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917296 5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917299 5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917308 5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917311 5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917314 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917317 5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:33:01.933536 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917320 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917322 5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917325 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917328 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917330 5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917335 5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917338 5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917341 5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917343 5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917346 5515 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917349 5515 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917352 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917354 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917357 5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917360 5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917364 5515 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917366 5515 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917369 5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917372 5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917375 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:33:01.934009 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917377 5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917380 5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917383 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917386 5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917389 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917391 5515 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917394 5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917397 5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917399 5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917407 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917412 5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917416 5515 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917419 5515 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917423 5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917425 5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917428 5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917431 5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917436 5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917438 5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917441 5515 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:33:01.934515 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917444 5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917447 5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917449 5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917453 5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917457 5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917460 5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917463 5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917468 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917470 5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917473 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917476 5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917478 5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917481 5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917484 5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917487 5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917494 5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917498 5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917500 5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917503 5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:33:01.935008 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917506 5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917508 5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:01.917511 5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.917573 5515 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.918703 5515 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.918763 5515 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.918943 5515 server.go:1014] "Starting client certificate rotation" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.919147 5515 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.919257 5515 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:26:50 +0000 UTC" deadline="2028-07-06 21:46:44.652179416 +0000 UTC" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.919325 5515 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14769h13m42.732856101s" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.919582 5515 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.920829 5515 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.922953 5515 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.926466 5515 log.go:25] "Validated CRI v1 image API" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.927361 5515 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:33:01.935518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.929525 5515 fs.go:135] Filesystem UUIDs: map[426bc9d2-8242-4677-a9d6-8316a32fca56:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d9ecd2b2-462d-47a8-a0bc-aca898527fa3:/dev/nvme0n1p3] Oct 30 12:33:01.935933 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.929548 5515 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/016579cdb87e2e798847a0e30caac0807a8cce0bc451d5b051322e10b3925939/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/016579cdb87e2e798847a0e30caac0807a8cce0bc451d5b051322e10b3925939/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05f5ed45a54acd8d5c1fa8f2b50fe8668f55272b0bbe72a2261538c618537a80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05f5ed45a54acd8d5c1fa8f2b50fe8668f55272b0bbe72a2261538c618537a80/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10551bd805ee44c030ec2cc0cc61050728fde9102f459b12c1aac64fea0f818b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10551bd805ee44c030ec2cc0cc61050728fde9102f459b12c1aac64fea0f818b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1827a3cf2e04b66e1352f2d22df1ad4000b8986b5ab9c0422431a7d2ae9cba6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1827a3cf2e04b66e1352f2d22df1ad4000b8986b5ab9c0422431a7d2ae9cba6b/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/457ceac783c6ae2395922e13d69810b584047f05bc64e8f4314d5d57e5fd4521/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/457ceac783c6ae2395922e13d69810b584047f05bc64e8f4314d5d57e5fd4521/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/649e56f7044427264966013bb115f28e137d727d9b61aecbe8dae8c1f405bfd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/649e56f7044427264966013bb115f28e137d727d9b61aecbe8dae8c1f405bfd1/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d965288c14cb8d369763f0f7b69aa1295310a7bcc9177dd32191a483e8987f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d965288c14cb8d369763f0f7b69aa1295310a7bcc9177dd32191a483e8987f9/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d91157ee3deb2e73b1b2f6c38bc780ac83b7de2d8b24f037209c4b1fa9de8dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d91157ee3deb2e73b1b2f6c38bc780ac83b7de2d8b24f037209c4b1fa9de8dc/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8475c423ff1416280b35d5a6d2685a3fcf4c351b98060e4cf12ebdc98196b3bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8475c423ff1416280b35d5a6d2685a3fcf4c351b98060e4cf12ebdc98196b3bc/userdata/shm major:0 minor:306 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aedbb4e9a0fdbed6117f4cb28a87473625265160767c63a17bab2a3b6b445cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aedbb4e9a0fdbed6117f4cb28a87473625265160767c63a17bab2a3b6b445cf/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bc47e92b400edaf7ac8f478ee6f2b859598b22530f61ee07f922ca67769c5d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bc47e92b400edaf7ac8f478ee6f2b859598b22530f61ee07f922ca67769c5d3/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dc19077288f4a9d95e9ee2d85b372c3c317e20239177e2863272fee2d42a83e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dc19077288f4a9d95e9ee2d85b372c3c317e20239177e2863272fee2d42a83e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9835506cea0d204c5020c73b46fc2b605165090ce6b34c2b0ddd23260117a198/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9835506cea0d204c5020c73b46fc2b605165090ce6b34c2b0ddd23260117a198/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a454ceec44bbe20b1030089ab27e07caf34ec226cb748bd76f202ca7d5bb554c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a454ceec44bbe20b1030089ab27e07caf34ec226cb748bd76f202ca7d5bb554c/userdata/shm major:0 minor:388 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6a3b5bb45b25d684802403358e3ab0025a146f34f27ea79a25dea6fa1eb1f51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6a3b5bb45b25d684802403358e3ab0025a146f34f27ea79a25dea6fa1eb1f51/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3e169ca1811e75f1bf7f490075a92fb2c451a96050cdc0f71386e4f2e19b79d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3e169ca1811e75f1bf7f490075a92fb2c451a96050cdc0f71386e4f2e19b79d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4a31801efaf1ac66875a037a1bc2e27d75b76501ad396656efcc28ea194fdc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4a31801efaf1ac66875a037a1bc2e27d75b76501ad396656efcc28ea194fdc8/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c97a632f2a75fa873d60632942041c9abb69af13a3cc44cc510b70c59783a85d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c97a632f2a75fa873d60632942041c9abb69af13a3cc44cc510b70c59783a85d/userdata/shm major:0 minor:391 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d149e2fb3ac56044cd991c4a991df3aa3a36e6cda1e1d80f2768fee0ccc96d90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d149e2fb3ac56044cd991c4a991df3aa3a36e6cda1e1d80f2768fee0ccc96d90/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d670e49694a2f90b60b06edb9289e84cdde05c46fcfc100f8f79d4648dee2fc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d670e49694a2f90b60b06edb9289e84cdde05c46fcfc100f8f79d4648dee2fc3/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddc1fb8a75fd76c11f1a215eda5e5ae2d9b47e2a6f3c37093187b03b32e133a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddc1fb8a75fd76c11f1a215eda5e5ae2d9b47e2a6f3c37093187b03b32e133a2/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb81b96d43e797d98f7600765712d20665be5ad2d50f77401d5129ebe09d8501/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb81b96d43e797d98f7600765712d20665be5ad2d50f77401d5129ebe09d8501/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0686737a-10d7-4761-877c-46bd3df261c2/volumes/kubernetes.io~projected/kube-api-access-vcs5g:{mountpoint:/var/lib/kubelet/pods/0686737a-10d7-4761-877c-46bd3df261c2/volumes/kubernetes.io~projected/kube-api-access-vcs5g major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0686737a-10d7-4761-877c-46bd3df261c2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0686737a-10d7-4761-877c-46bd3df261c2/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/102a0155-f391-4c1e-98d7-eea36a51615e/volumes/kubernetes.io~projected/kube-api-access-lbk6q:{mountpoint:/var/lib/kubelet/pods/102a0155-f391-4c1e-98d7-eea36a51615e/volumes/kubernetes.io~projected/kube-api-access-lbk6q major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/102a0155-f391-4c1e-98d7-eea36a51615e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/102a0155-f391-4c1e-98d7-eea36a51615e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38/volumes/kubernetes.io~projected/kube-api-access-rc5ld:{mountpoint:/var/lib/kubelet/pods/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38/volumes/kubernetes.io~projected/kube-api-access-rc5ld major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24b12750-62c7-478f-bea4-8aee763363f3/volumes/kubernetes.io~projected/kube-api-access-jmkcs:{mountpoint:/var/lib/kubelet/pods/24b12750-62c7-478f-bea4-8aee763363f3/volumes/kubernetes.io~projected/kube-api-access-jmkcs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/475cbe1a-bcd8-4575-9c39-3c3394f062e4/volumes/kubernetes.io~projected/kube-api-access-j8jr9:{mountpoint:/var/lib/kubelet/pods/475cbe1a-bcd8-4575-9c39-3c3394f062e4/volumes/kubernetes.io~projected/kube-api-access-j8jr9 major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/475cbe1a-bcd8-4575-9c39-3c3394f062e4/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/475cbe1a-bcd8-4575-9c39-3c3394f062e4/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/web-config/alertmanager/9 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/web-config/config-reloader/6 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~empty-dir/config-out major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~projected/kube-api-access-ddf98:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~projected/kube-api-access-ddf98 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~projected/tls-assets major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/config-volume major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/web-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/532d8fee-b407-48ed-b9f5-77b8ad7f7630/volumes/kubernetes.io~projected/kube-api-access-nhkph:{mountpoint:/var/lib/kubelet/pods/532d8fee-b407-48ed-b9f5-77b8ad7f7630/volumes/kubernetes.io~projected/kube-api-access-nhkph major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/532d8fee-b407-48ed-b9f5-77b8ad7f7630/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/532d8fee-b407-48ed-b9f5-77b8ad7f7630/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/532d8fee-b407-48ed-b9f5-77b8ad7f7630/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/532d8fee-b407-48ed-b9f5-77b8ad7f7630/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b35dcc8-6a2f-4cc9-a810-85ec056b5899/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5b35dcc8-6a2f-4cc9-a810-85ec056b5899/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b73fd1c-6f57-4172-b98d-c8b8330efd83/volumes/kubernetes.io~projected/kube-api-access-sdpzv:{mountpoint:/var/lib/kubelet/pods/5b73fd1c-6f57-4172-b98d-c8b8330efd83/volumes/kubernetes.io~projected/kube-api-access-sdpzv major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b73fd1c-6f57-4172-b98d-c8b8330efd83/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5b73fd1c-6f57-4172-b98d-c8b8330efd83/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b73fd1c-6f57-4172-b98d-c8b8330efd83/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/5b73fd1c-6f57-4172-b98d-c8b8330efd83/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f15c7b4-5807-4438-81c7-5a392eca8b2a/volumes/kubernetes.io~projected/kube-api-access-6crm9:{mountpoint:/var/lib/kubelet/pods/5f15c7b4-5807-4438-81c7-5a392eca8b2a/volumes/kubernetes.io~projected/kube-api-access-6crm9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88627b5e-75be-4029-ba55-51759aedc92d/volumes/kubernetes.io~projected/kube-api-access-bt5zc:{mountpoint:/var/lib/kubelet/pods/88627b5e-75be-4029-ba55-51759aedc92d/volumes/kubernetes.io~projected/kube-api-access-bt5zc major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes/kubernetes.io~projected/kube-api-access-vg58c:{mountpoint:/var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes/kubernetes.io~projected/kube-api-access-vg58c major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77/volumes/kubernetes.io~projected/kube-api-access-hr2k6:{mountpoint:/var/lib/kubelet/pods/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77/volumes/kubernetes.io~projected/kube-api-access-hr2k6 major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5f156b-c309-4b4b-8a77-270faeb79648/volumes/kubernetes.io~projected/kube-api-access-cpc4l:{mountpoint:/var/lib/kubelet/pods/db5f156b-c309-4b4b-8a77-270faeb79648/volumes/kubernetes.io~projected/kube-api-access-cpc4l major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db5f156b-c309-4b4b-8a77-270faeb79648/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/db5f156b-c309-4b4b-8a77-270faeb79648/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~projected/kube-api-access-8r7rm:{mountpoint:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~projected/kube-api-access-8r7rm major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc939343-5581-480f-8c67-37ba0d1188c6/volumes/kubernetes.io~projected/kube-api-access-q5l8w:{mountpoint:/var/lib/kubelet/pods/dc939343-5581-480f-8c67-37ba0d1188c6/volumes/kubernetes.io~projected/kube-api-access-q5l8w major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc939343-5581-480f-8c67-37ba0d1188c6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/dc939343-5581-480f-8c67-37ba0d1188c6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e218ebda-caba-45d2-963a-77b96b0af90f/volumes/kubernetes.io~projected/kube-api-access-7bhwv:{mountpoint:/var/lib/kubelet/pods/e218ebda-caba-45d2-963a-77b96b0af90f/volumes/kubernetes.io~projected/kube-api-access-7bhwv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3354c62-d061-4f12-9c22-b21455da0faf/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e3354c62-d061-4f12-9c22-b21455da0faf/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3354c62-d061-4f12-9c22-b21455da0faf/volumes/kubernetes.io~projected/kube-api-access-pqzgz:{mountpoint:/var/lib/kubelet/pods/e3354c62-d061-4f12-9c22-b21455da0faf/volumes/kubernetes.io~projected/kube-api-access-pqzgz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3354c62-d061-4f12-9c22-b21455da0faf/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e3354c62-d061-4f12-9c22-b21455da0faf/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0/volumes/kubernetes.io~projected/kube-api-access-d2b7j:{mountpoint:/var/lib/kubelet/pods/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0/volumes/kubernetes.io~projected/kube-api-access-d2b7j major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecc1b4c7-ef74-4105-b613-2f53486e7aad/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ecc1b4c7-ef74-4105-b613-2f53486e7aad/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2e3b448-9d0e-4141-a929-72732655d497/volumes/kubernetes.io~projected/kube-api-access-g68t4:{mountpoint:/var/lib/kubelet/pods/f2e3b448-9d0e-4141-a929-72732655d497/volumes/kubernetes.io~projected/kube-api-access-g68t4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb075adb-3219-4e67-a61a-00a6a422eef7/volumes/kubernetes.io~projected/kube-api-access-5m58t:{mountpoint:/var/lib/kubelet/pods/fb075adb-3219-4e67-a61a-00a6a422eef7/volumes/kubernetes.io~projected/kube-api-access-5m58t major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a74ec467e463fe31bdd7361c454b4d21274b2ebe24ddbd8ad2a4b4ddef070a93/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/077694c7e8f37402d328c968139c36406fa5a051ec633612d5475a3db3fba2e5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/acf8f29b8a476deba9531d3cb8ec14133578296a9948bd9f5b0c2b91ec2a84d6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b47be5104c93936f936028f394b228937d393ce55970fb95fd76a6b3c3c63054/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b51c197f376c48879e444f021e4575cc2ef82cb03c0e0ab4fcc2e7f36b128c15/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fc17250d3a0f6916c464167f2f40a97be236854dc13af1e570e060f59d45c9e6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6005a7be3933df5c98de1b5d979504e2dd85891f56610e6c9917dfb11900d34f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2759cc915d297c656a47cb804f80e9e68160315f9e5d7d95a6493edeadb102c0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/18d47d5b1012bb7e5740f0ba34ae5c9b0bf74a983b2ce6a042c30f347de700c4/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/8d1e1387d8175d118bffbb010e4ddeaa25a06df74ef773da1f447ddf9f135e79/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/b1b7bbe74cef16f09e4fa1613bf411fff5a04f617f525e55fbe2d30cd2dda384/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/8286b68f3e264f9b250307ec1ef96428c56c18499222287c6c649710544c66f9/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4d9606870a0d4c500bb4b1e3477d9e6efa386812cd20bd90ec5bf59b98971aa9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8414e3e9d9450d4b8c1ff6b080b82bba8cd6364bf8f49dbb624300e37b55d35e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2a70f18f30f542e5b58f46601db0044124355b2e531f9df6fd25fd779fa2e11f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4b8d12598a58ee92c2a05b533cb9426a16ae2c2d2d66821eb6b57a6dda4689ff/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f12fc4df5e62370678e2b1964d19b1073c6d4b0c6980be6d7296375f815f7038/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f3b41c691b8f3398d8f29a8a8489396c982a20fc96fb90a03543875ae7a8e4dd/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f051f77a18b004ab1457e2afcd0b0b92f3d68acc2ddb8909af240c620d6a0af2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c91ff8ac257a582719b54a841a9d30c7bd7dd37b89690682b34e289c53f6b28b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4c24783d036a9d6a8d2439cac24679026c3094c1ae75ec12042d6a0275cd741c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/ce17db808f5297b34be91d3fb1622c1abdf8d325ca512b201a701240f1f9f28d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/baa39b70667c92663ae17c8758dbf9fd12617c78555cdfe5af0a7bd2b93a1503/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/0f60397b097a3b2f9b4c94afef53bc8b5c2b275b06dd63e3a788cd62e745ff05/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/364c9a80fb7f672c14ed4f720b1a2531c6e69e2db3c91d191e7d9127945ad801/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/03194f8410965aaa67ca6b867f6b939950c9b1e350947a2a68c4463f066af666/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/30b374e37f291ecdec0cda2552e4eebf146f62a98067a05ba2bd2ef57593b970/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/62b2bba3fe4f7fc1f970754aeb3a1dcb8b3bc1ad8a229dfae494dbdf893b8f31/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/6cd3c4a6978f1905476f9d63700baa90eb8a5faa5aa2c1ab1e4f75ff6c2fd196/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/0a4cd34b47293ea6281511dcc8a98821943908ed09c27c500d1cf03acda0920e/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b789b16e75638ff8715b34b6297c02b086eef8444db2aa420e0556888ee3d905/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/2f29bcdb19e4a906d8899c7f0e3c330434c3f2f2057c53185e4055e57de87c4e/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/3122e6f2e43f6b1fe3c923a5e98797d56b87414d2ac5d409700d5c2ae432e606/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/5c7e02572239e09ea2f160c0d82cbdc5f23b1087236d6af5ad00f6965f2d0413/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/337f2a7df57dd4976b43b12db87b74e5f285f36d411fd9138d3d00ed9bbfeac7/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/4dc2a1a72870cd131f9f0e3ecd771848cf9e8fa5a93ccf2a414371de58be72ac/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/234ef4016ec7b5ef394f29ded09963b7bfc155cfcda65a02f2eb69bf85369f0f/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/e1f454d83e5d6728fbdec6eb855e0a5affb67eb3e0956c3c4797bd1290c5f4fb/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a2cbe57e1964bbad2939569b1a8a0924860cd000ae59e57330c4f8cf9891f730/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/41d330ea444f9c9427083d11db5dbe90619ebfc42a9c4aa91c2515dd8ab2d86e/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/994bbc14697b0f6107f9a1605db2a7bc8f955920adb956b14e12e544fb1e9b3d/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/a48ed5b064163b390a6533b0bda9e8c355216b7de6d3c982e7ffacbfc760f1e3/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/6ecdc52222af618590f643d1b3b8595633ab53c5eb024ba0345707d30f1cf377/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/329cf3136fcf1e73b93d83b15278a8044f386fdf45aa1a99b5b0fdcb8224f43e/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/a5204a8582aff225211d5122c43aa52596fbe67d8c1f69ebf1724f71b83732e2/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/2b5019a7729c4b9e6eee4888ef2d7e92cc68fcf84aa252a5f4c89e4a314419d9/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/95b6174697e369ec2cfafab7eb55f7ba5e42d3f9e975aac4361f9e29a90e237a/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/81797bcaecf839ab54e7066dd5046ad612404dfc52b23e2ee09ecf1e295170a2/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/35822ad9abf8192a4c35517d160ce65a5e4b729e3862c9ee0ce7e429ce511a8c/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/bcd8cb698117d264e1d9dffb1055f644a633c568971519f2117a7592295824d3/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/46ffafc9f4205475258af75600df1ac25c8f8a10d5a6bf12db96ded63426e338/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/f0133e99b5acabc13a7234cf6e84e6c38df98fd535cbad73f923f3c467677faa/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/8903d3147c4593fdf840a4f9773f57878dc969a3bb3dad595264498faea55637/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/4f57b99916b83090503f18deaf0905e0c5501286a0d49850564214435ccae55b/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/9494bc4574ea45f01377c89b35405c4d8eed36909cbc667b91732cf0d2d854b9/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/cc087363ce2bcfe8085eec1cff83b3b50fdf102725967ed61a062b6b21ae4a58/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/18c4437432836beb344801f124e0e4e71b776f20792d48a1dde69f1908ee3eed/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/457f883ac9595c84eb2a296c1f9b4df24a655da9fcdf3192af3ef4265d2239ce/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/434a9a05bb423ba4301c55e817ef4a3f31f44fb665026272b731e8cc298ff821/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/106ad972a61ff8a932ddf2624ccee5315432a8f6f302a2212520732a8f99ec02/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/53d40c45c4c765244fad04e7727ed9f01f8f8c6801936b7373fd1843d29e08f8/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/fe589ea7b7c4189b8b0f5dce7e2d3ae67ea10ec7a4ee6f089de9d34f20f83555/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/eb59ef3265993672f6de2572387312ad5ba2c9c4c3de7d2dc552f906b65a6a82/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/88ac33185f5bde4b7ebae9d1ce4d10dbd92f3ab0ecd1eea5eaf292d3160987ba/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/dbd4ab4b27318f14106e3f6f4f64a4b0ac11f70a29c84dcf3ecc58743459faa6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/9452244596b710e54fa6abab667067bddcf4be72fc0c2f2d4487432524722e77/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ebfcb3440c0209d6c5c4834ae628bbb658fc219764f1742c4e375e18ce4f680b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/542d02c460f282896498379b51a802664459386abee0c0119b82f8e4c39eaf66/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c20732d107f71214ca39c247d1e5fa4753d3464be7fe412dff3e6fb1c38fc7b0/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b94e11a1a316e0ff6db80529c43428d6e63fe52dfdb73accd4715246392b4a71/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 12:33:01.937548 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.937153 5515 manager.go:217] Machine: {Timestamp:2026-10-30 12:33:01.936646353 +0000 UTC m=+0.063447243 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126726 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f441c8d90bb88daaa94aca0132be SystemUUID:ec24f441-c8d9-0bb8-8daa-a94aca0132be BootID:345f4e25-b980-422f-aec4-168548d2d0eb Filesystems:[{Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:428 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3354c62-d061-4f12-9c22-b21455da0faf/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24b12750-62c7-478f-bea4-8aee763363f3/volumes/kubernetes.io~projected/kube-api-access-jmkcs DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05f5ed45a54acd8d5c1fa8f2b50fe8668f55272b0bbe72a2261538c618537a80/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6a3b5bb45b25d684802403358e3ab0025a146f34f27ea79a25dea6fa1eb1f51/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0686737a-10d7-4761-877c-46bd3df261c2/volumes/kubernetes.io~projected/kube-api-access-vcs5g DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d965288c14cb8d369763f0f7b69aa1295310a7bcc9177dd32191a483e8987f9/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/102a0155-f391-4c1e-98d7-eea36a51615e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:380 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88627b5e-75be-4029-ba55-51759aedc92d/volumes/kubernetes.io~projected/kube-api-access-bt5zc DeviceMajor:0 DeviceMinor:390 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc939343-5581-480f-8c67-37ba0d1188c6/volumes/kubernetes.io~projected/kube-api-access-q5l8w DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d670e49694a2f90b60b06edb9289e84cdde05c46fcfc100f8f79d4648dee2fc3/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c97a632f2a75fa873d60632942041c9abb69af13a3cc44cc510b70c59783a85d/userdata/shm DeviceMajor:0 DeviceMinor:391 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d149e2fb3ac56044cd991c4a991df3aa3a36e6cda1e1d80f2768fee0ccc96d90/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0/volumes/kubernetes.io~projected/kube-api-access-d2b7j DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e3354c62-d061-4f12-9c22-b21455da0faf/volumes/kubernetes.io~projected/kube-api-access-pqzgz DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b73fd1c-6f57-4172-b98d-c8b8330efd83/volumes/kubernetes.io~projected/kube-api-access-sdpzv DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8475c423ff1416280b35d5a6d2685a3fcf4c351b98060e4cf12ebdc98196b3bc/userdata/shm DeviceMajor:0 DeviceMinor:306 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/475cbe1a-bcd8-4575-9c39-3c3394f062e4/volumes/kubernetes.io~projected/kube-api-access-j8jr9 DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4a31801efaf1ac66875a037a1bc2e27d75b76501ad396656efcc28ea194fdc8/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0686737a-10d7-4761-877c-46bd3df261c2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc939343-5581-480f-8c67-37ba0d1188c6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b73fd1c-6f57-4172-b98d-c8b8330efd83/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/102a0155-f391-4c1e-98d7-eea36a51615e/volumes/kubernetes.io~projected/kube-api-access-lbk6q DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b35dcc8-6a2f-4cc9-a810-85ec056b5899/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/532d8fee-b407-48ed-b9f5-77b8ad7f7630/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9835506cea0d204c5020c73b46fc2b605165090ce6b34c2b0ddd23260117a198/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~projected/kube-api-access-8r7rm DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3e169ca1811e75f1bf7f490075a92fb2c451a96050cdc0f71386e4f2e19b79d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:423 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aedbb4e9a0fdbed6117f4cb28a87473625265160767c63a17bab2a3b6b445cf/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10551bd805ee44c030ec2cc0cc61050728fde9102f459b12c1aac64fea0f818b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d91157ee3deb2e73b1b2f6c38bc780ac83b7de2d8b24f037209c4b1fa9de8dc/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddc1fb8a75fd76c11f1a215eda5e5ae2d9b47e2a6f3c37093187b03b32e133a2/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f15c7b4-5807-4438-81c7-5a392eca8b2a/volumes/kubernetes.io~projected/kube-api-access-6crm9 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bc47e92b400edaf7ac8f478ee6f2b859598b22530f61ee07f922ca67769c5d3/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/016579cdb87e2e798847a0e30caac0807a8cce0bc451d5b051322e10b3925939/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/457ceac783c6ae2395922e13d69810b584047f05bc64e8f4314d5d57e5fd4521/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b73fd1c-6f57-4172-b98d-c8b8330efd83/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a454ceec44bbe20b1030089ab27e07caf34ec226cb748bd76f202ca7d5bb554c/userdata/shm DeviceMajor:0 DeviceMinor:388 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77/volumes/kubernetes.io~projected/kube-api-access-hr2k6 DeviceMajor:0 DeviceMinor:402 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1827a3cf2e04b66e1352f2d22df1ad4000b8986b5ab9c0422431a7d2ae9cba6b/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/db5f156b-c309-4b4b-8a77-270faeb79648/volumes/kubernetes.io~projected/kube-api-access-cpc4l DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/475cbe1a-bcd8-4575-9c39-3c3394f062e4/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/532d8fee-b407-48ed-b9f5-77b8ad7f7630/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecc1b4c7-ef74-4105-b613-2f53486e7aad/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:385 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbafdcd8-268f-4128-b17e-10d3d92e2227/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/532d8fee-b407-48ed-b9f5-77b8ad7f7630/volumes/kubernetes.io~projected/kube-api-access-nhkph DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dc19077288f4a9d95e9ee2d85b372c3c317e20239177e2863272fee2d42a83e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e218ebda-caba-45d2-963a-77b96b0af90f/volumes/kubernetes.io~projected/kube-api-access-7bhwv DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb81b96d43e797d98f7600765712d20665be5ad2d50f77401d5129ebe09d8501/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3354c62-d061-4f12-9c22-b21455da0faf/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/649e56f7044427264966013bb115f28e137d727d9b61aecbe8dae8c1f405bfd1/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes/kubernetes.io~projected/kube-api-access-vg58c DeviceMajor:0 DeviceMinor:429 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38/volumes/kubernetes.io~projected/kube-api-access-rc5ld DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db5f156b-c309-4b4b-8a77-270faeb79648/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~projected/kube-api-access-ddf98 DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb075adb-3219-4e67-a61a-00a6a422eef7/volumes/kubernetes.io~projected/kube-api-access-5m58t DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f2e3b448-9d0e-4141-a929-72732655d497/volumes/kubernetes.io~projected/kube-api-access-g68t4 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:016579cdb87e2e7 MacAddress:8a:6b:a1:22:8b:6c Speed:10000 Mtu:8901} {Name:1827a3cf2e04b66 MacAddress:52:47:a1:e6:44:98 Speed:10000 Mtu:8901} {Name:457ceac783c6ae2 MacAddress:32:c0:61:48:00:77 Speed:10000 Mtu:8901} {Name:649e56f70444272 MacAddress:aa:61:49:61:40:7e Speed:10000 Mtu:8901} {Name:6d965288c14cb8d MacAddress:9a:cd:6c:6f:34:6c Speed:10000 Mtu:8901} {Name:7d91157ee3deb2e MacAddress:fa:8d:f1:b0:c7:a3 Speed:10000 Mtu:8901} {Name:8475c423ff14162 MacAddress:32:76:a4:14:f5:76 Speed:10000 Mtu:8901} {Name:9835506cea0d204 MacAddress:e2:75:a3:d2:c3:3a Speed:10000 Mtu:8901} {Name:a454ceec44bbe20 MacAddress:d6:f5:d7:7d:1d:75 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c4:5e:29:0f Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:c4:43:4a:a2:01 Speed:0 Mtu:8901} {Name:c97a632f2a75fa8 MacAddress:de:28:80:d1:21:e2 Speed:10000 Mtu:8901} {Name:ddc1fb8a75fd76c MacAddress:56:61:c4:84:c6:b0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c4:5e:29:0f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:96:6f:db:fd:79:aa Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:02:af:58:b7:95:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:33:01.937548 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.937552 5515 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:33:01.937882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.937604 5515 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:33:01.937914 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.937890 5515 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:33:01.938067 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.937912 5515 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:33:01.938113 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938069 5515 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:33:01.938113 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938076 5515 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:33:01.938113 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938090 5515 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:33:01.938113 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938110 5515 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:33:01.938218 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938209 5515 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:33:01.938412 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938401 5515 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:33:01.938529 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938520 5515 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:33:01.938569 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938560 5515 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:33:01.938599 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938579 5515 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:33:01.938599 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938590 5515 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:33:01.938679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938601 5515 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:33:01.938679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.938609 5515 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:33:01.939690 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.939674 5515 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:33:01.939745 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.939692 5515 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:33:01.940357 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.940320 5515 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:33:01.952174 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952150 5515 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:33:01.952174 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952168 5515 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:33:01.952504 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952313 5515 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 12:33:01.952546 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952522 5515 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:33:01.952579 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952550 5515 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:33:01.952661 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952649 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:33:01.952696 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952671 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:33:01.952696 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952684 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:33:01.952696 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952692 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952698 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952704 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952711 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952716 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952723 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952729 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952741 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952753 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952780 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:33:01.952784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.952789 5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:33:01.956472 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.956424 5515 server.go:1263] "Started kubelet" Oct 30 12:33:01.956583 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.956499 5515 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:33:01.958747 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.956538 5515 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:33:01.958747 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.956760 5515 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:33:01.958747 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.957076 5515 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:33:01.958747 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.957986 5515 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:33:01.957006 ip-10-0-134-193 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:33:01.960020 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.960004 5515 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:33:01.960134 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.960057 5515 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:33:01.960214 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.960196 5515 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:33:01.960214 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.960208 5515 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:33:01.960373 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.960218 5515 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:26:52 +0000 UTC" deadline="2028-05-01 08:29:51.976052606 +0000 UTC" Oct 30 12:33:01.960373 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.960254 5515 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13171h56m50.015801968s" Oct 30 12:33:01.960373 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.960307 5515 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:33:01.960868 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:01.960850 5515 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-193.ec2.internal\" not found" Oct 30 12:33:01.964125 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.961667 5515 factory.go:55] Registering systemd factory Oct 30 12:33:01.964125 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.962524 5515 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:33:01.964125 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.964114 5515 factory.go:153] Registering CRI-O factory Oct 30 12:33:01.965453 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.964131 5515 factory.go:223] Registration of the crio container factory successfully Oct 30 12:33:01.965453 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.964217 5515 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:33:01.965453 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.964237 5515 factory.go:103] Registering Raw factory Oct 30 12:33:01.965453 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.964255 5515 manager.go:1199] Started watching for new ooms in manager Oct 30 12:33:01.965453 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.964624 5515 manager.go:319] Starting recovery of all containers Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.968968 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0686737a-10d7-4761-877c-46bd3df261c2" volumeName="kubernetes.io/projected/0686737a-10d7-4761-877c-46bd3df261c2-kube-api-access-vcs5g" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969041 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="475cbe1a-bcd8-4575-9c39-3c3394f062e4" volumeName="kubernetes.io/projected/475cbe1a-bcd8-4575-9c39-3c3394f062e4-kube-api-access-j8jr9" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969057 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="475cbe1a-bcd8-4575-9c39-3c3394f062e4" volumeName="kubernetes.io/secret/475cbe1a-bcd8-4575-9c39-3c3394f062e4-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969072 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532d8fee-b407-48ed-b9f5-77b8ad7f7630" volumeName="kubernetes.io/projected/532d8fee-b407-48ed-b9f5-77b8ad7f7630-kube-api-access-nhkph" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969085 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8c4da3-8154-4b51-ac73-49afe96db85e" volumeName="kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969096 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8c4da3-8154-4b51-ac73-49afe96db85e" volumeName="kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969109 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb075adb-3219-4e67-a61a-00a6a422eef7" volumeName="kubernetes.io/projected/fb075adb-3219-4e67-a61a-00a6a422eef7-kube-api-access-5m58t" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969122 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="102a0155-f391-4c1e-98d7-eea36a51615e" volumeName="kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969139 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38" volumeName="kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-daemon-config" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969151 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969164 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969176 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969188 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3354c62-d061-4f12-9c22-b21455da0faf" volumeName="kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-env-overrides" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969203 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969219 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969233 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8c4da3-8154-4b51-ac73-49afe96db85e" volumeName="kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969245 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="983c8f5c-68cf-42a4-b931-4ceb2b7e1e77" volumeName="kubernetes.io/empty-dir/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-tmp" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969257 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5f156b-c309-4b4b-8a77-270faeb79648" volumeName="kubernetes.io/empty-dir/db5f156b-c309-4b4b-8a77-270faeb79648-data-volume" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969288 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38" volumeName="kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cni-binary-copy" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969305 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24b12750-62c7-478f-bea4-8aee763363f3" volumeName="kubernetes.io/empty-dir/24b12750-62c7-478f-bea4-8aee763363f3-tmp-dir" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969316 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f15c7b4-5807-4438-81c7-5a392eca8b2a" volumeName="kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969328 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbafdcd8-268f-4128-b17e-10d3d92e2227" volumeName="kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-grpc-tls" seLinuxMountContext="" Oct 30 12:33:01.969334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969347 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="102a0155-f391-4c1e-98d7-eea36a51615e" volumeName="kubernetes.io/projected/102a0155-f391-4c1e-98d7-eea36a51615e-kube-api-access-lbk6q" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969359 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88627b5e-75be-4029-ba55-51759aedc92d" volumeName="kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969370 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8c4da3-8154-4b51-ac73-49afe96db85e" volumeName="kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969383 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e218ebda-caba-45d2-963a-77b96b0af90f" volumeName="kubernetes.io/configmap/e218ebda-caba-45d2-963a-77b96b0af90f-serviceca" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969394 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e218ebda-caba-45d2-963a-77b96b0af90f" volumeName="kubernetes.io/projected/e218ebda-caba-45d2-963a-77b96b0af90f-kube-api-access-7bhwv" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969406 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3354c62-d061-4f12-9c22-b21455da0faf" volumeName="kubernetes.io/projected/e3354c62-d061-4f12-9c22-b21455da0faf-kube-api-access-pqzgz" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969423 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0" volumeName="kubernetes.io/projected/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-kube-api-access-d2b7j" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969437 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969449 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5f156b-c309-4b4b-8a77-270faeb79648" volumeName="kubernetes.io/projected/db5f156b-c309-4b4b-8a77-270faeb79648-kube-api-access-cpc4l" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969468 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5f156b-c309-4b4b-8a77-270faeb79648" volumeName="kubernetes.io/secret/db5f156b-c309-4b4b-8a77-270faeb79648-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969484 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc939343-5581-480f-8c67-37ba0d1188c6" volumeName="kubernetes.io/configmap/dc939343-5581-480f-8c67-37ba0d1188c6-config-volume" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969495 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b73fd1c-6f57-4172-b98d-c8b8330efd83" volumeName="kubernetes.io/projected/5b73fd1c-6f57-4172-b98d-c8b8330efd83-kube-api-access-sdpzv" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969508 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b73fd1c-6f57-4172-b98d-c8b8330efd83" volumeName="kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969519 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f15c7b4-5807-4438-81c7-5a392eca8b2a" volumeName="kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969532 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="983c8f5c-68cf-42a4-b931-4ceb2b7e1e77" volumeName="kubernetes.io/projected/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-kube-api-access-hr2k6" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969545 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbafdcd8-268f-4128-b17e-10d3d92e2227" volumeName="kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969557 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc939343-5581-480f-8c67-37ba0d1188c6" volumeName="kubernetes.io/projected/dc939343-5581-480f-8c67-37ba0d1188c6-kube-api-access-q5l8w" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969569 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc939343-5581-480f-8c67-37ba0d1188c6" volumeName="kubernetes.io/secret/dc939343-5581-480f-8c67-37ba0d1188c6-metrics-tls" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969582 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3354c62-d061-4f12-9c22-b21455da0faf" volumeName="kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-script-lib" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969593 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecc1b4c7-ef74-4105-b613-2f53486e7aad" volumeName="kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969604 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2e3b448-9d0e-4141-a929-72732655d497" volumeName="kubernetes.io/configmap/f2e3b448-9d0e-4141-a929-72732655d497-iptables-alerter-script" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969617 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2e3b448-9d0e-4141-a929-72732655d497" volumeName="kubernetes.io/projected/f2e3b448-9d0e-4141-a929-72732655d497-kube-api-access-g68t4" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969630 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532d8fee-b407-48ed-b9f5-77b8ad7f7630" volumeName="kubernetes.io/configmap/532d8fee-b407-48ed-b9f5-77b8ad7f7630-metrics-client-ca" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969643 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532d8fee-b407-48ed-b9f5-77b8ad7f7630" volumeName="kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969654 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b35dcc8-6a2f-4cc9-a810-85ec056b5899" volumeName="kubernetes.io/configmap/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-konnectivity-ca" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969665 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b73fd1c-6f57-4172-b98d-c8b8330efd83" volumeName="kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969677 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f15c7b4-5807-4438-81c7-5a392eca8b2a" volumeName="kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-binary-copy" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969692 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc939343-5581-480f-8c67-37ba0d1188c6" volumeName="kubernetes.io/empty-dir/dc939343-5581-480f-8c67-37ba0d1188c6-tmp-dir" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969703 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969714 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969728 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="532d8fee-b407-48ed-b9f5-77b8ad7f7630" volumeName="kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-tls" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969741 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b73fd1c-6f57-4172-b98d-c8b8330efd83" volumeName="kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-metrics-client-ca" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969754 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b73fd1c-6f57-4172-b98d-c8b8330efd83" volumeName="kubernetes.io/empty-dir/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-textfile" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969767 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbafdcd8-268f-4128-b17e-10d3d92e2227" volumeName="kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969790 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbafdcd8-268f-4128-b17e-10d3d92e2227" volumeName="kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969804 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969816 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969827 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0686737a-10d7-4761-877c-46bd3df261c2" volumeName="kubernetes.io/secret/0686737a-10d7-4761-877c-46bd3df261c2-cert" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969842 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b35dcc8-6a2f-4cc9-a810-85ec056b5899" volumeName="kubernetes.io/secret/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-agent-certs" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969856 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8c4da3-8154-4b51-ac73-49afe96db85e" volumeName="kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969868 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38" volumeName="kubernetes.io/projected/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-kube-api-access-rc5ld" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969880 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969891 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f15c7b4-5807-4438-81c7-5a392eca8b2a" volumeName="kubernetes.io/projected/5f15c7b4-5807-4438-81c7-5a392eca8b2a-kube-api-access-6crm9" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969903 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbafdcd8-268f-4128-b17e-10d3d92e2227" volumeName="kubernetes.io/configmap/dbafdcd8-268f-4128-b17e-10d3d92e2227-metrics-client-ca" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969915 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24b12750-62c7-478f-bea4-8aee763363f3" volumeName="kubernetes.io/projected/24b12750-62c7-478f-bea4-8aee763363f3-kube-api-access-jmkcs" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969944 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b73fd1c-6f57-4172-b98d-c8b8330efd83" volumeName="kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-tls" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969958 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8c4da3-8154-4b51-ac73-49afe96db85e" volumeName="kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969970 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db5f156b-c309-4b4b-8a77-270faeb79648" volumeName="kubernetes.io/configmap/db5f156b-c309-4b4b-8a77-270faeb79648-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969982 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbafdcd8-268f-4128-b17e-10d3d92e2227" volumeName="kubernetes.io/projected/dbafdcd8-268f-4128-b17e-10d3d92e2227-kube-api-access-8r7rm" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.969993 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbafdcd8-268f-4128-b17e-10d3d92e2227" volumeName="kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970007 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3354c62-d061-4f12-9c22-b21455da0faf" volumeName="kubernetes.io/secret/e3354c62-d061-4f12-9c22-b21455da0faf-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970019 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970032 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea59390-80c3-432c-bda5-88af53de9263" volumeName="kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970045 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8c4da3-8154-4b51-ac73-49afe96db85e" volumeName="kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970056 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbafdcd8-268f-4128-b17e-10d3d92e2227" volumeName="kubernetes.io/empty-dir/dbafdcd8-268f-4128-b17e-10d3d92e2227-tmp" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970067 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbafdcd8-268f-4128-b17e-10d3d92e2227" volumeName="kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970079 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3354c62-d061-4f12-9c22-b21455da0faf" volumeName="kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-config" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970110 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0" volumeName="kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-tuned" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970124 5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0" volumeName="kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-tmp" seLinuxMountContext="" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970136 5515 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:33:01.970487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.970144 5515 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:33:01.991929 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.991883 5515 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:33:01.993202 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.993175 5515 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:33:01.993202 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.993196 5515 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:33:01.993431 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:01.993215 5515 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:33:01.993431 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:01.993257 5515 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 12:33:02.004369 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.004336 5515 generic.go:356] "Generic (PLEG): container finished" podID="4ea59390-80c3-432c-bda5-88af53de9263" containerID="4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024" exitCode=0 Oct 30 12:33:02.005590 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.005566 5515 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:33:02.005731 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.005672 5515 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:33:02.005800 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.005754 5515 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 12:33:02.009977 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.009944 5515 generic.go:356] "Generic (PLEG): container finished" podID="5b73fd1c-6f57-4172-b98d-c8b8330efd83" containerID="52105d3443d20f4855ad1b3f5281c7acfc59c76b68bcbebbd51d6f0145d18a2e" exitCode=0 Oct 30 12:33:02.018005 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.017973 5515 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="bc8d2b2d7e5cbfe6b4eca891ae926062498450f5fee4c0b737a3ad20f32a49c3" exitCode=0 Oct 30 12:33:02.018005 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.017998 5515 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="e4fbf7e79aea63c2f98239bf266f2330c80f64c8410fa3ed9b90f8d1a50de61f" exitCode=0 Oct 30 12:33:02.018005 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.018006 5515 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="52761aec52d8ec2d1cf12981554224e152420cd8696d43dfc28e14145aa493ab" exitCode=0 Oct 30 12:33:02.018005 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.018011 5515 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="bc14715900b2d796449387e601a3dd97016d4b68c868d9e30f0419677fb995c7" exitCode=0 Oct 30 12:33:02.018005 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.018016 5515 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="a1db3ebb83f75eebfefe37bcb00f63ca31ca102e376d0a5128a32fa9017da99c" exitCode=0 Oct 30 12:33:02.018005 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.018021 5515 generic.go:356] "Generic (PLEG): container finished" podID="5f15c7b4-5807-4438-81c7-5a392eca8b2a" containerID="68eafaa16eb579e675e0199706aec52ca5896460f4695d631df5e66ec094660d" exitCode=0 Oct 30 12:33:02.026367 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.026340 5515 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:33:02.026516 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.026376 5515 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:33:02.052094 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.052053 5515 generic.go:356] "Generic (PLEG): container finished" podID="e707ecff82755bdfb3889f9c2a2af067" containerID="b16c01813740e3d093b3d0198934043a5eb0fa8d1b65f621ce1b7b5478b2ba18" exitCode=0 Oct 30 12:33:02.070747 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.070723 5515 manager.go:324] Recovery completed Oct 30 12:33:02.092781 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.092743 5515 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:33:02.092781 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.092774 5515 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:33:02.092781 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.092798 5515 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:33:02.093051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.092951 5515 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 12:33:02.093051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.092967 5515 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 12:33:02.093051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.092984 5515 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 12:33:02.093051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.092990 5515 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 12:33:02.093051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.092995 5515 policy_none.go:50] "Start" Oct 30 12:33:02.093051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.093003 5515 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:33:02.093051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.093013 5515 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:33:02.093232 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.093131 5515 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 12:33:02.093232 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.093141 5515 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 12:33:02.093232 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.093149 5515 policy_none.go:44] "Start" Oct 30 12:33:02.093348 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:02.093329 5515 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 12:33:02.098411 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.097867 5515 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:33:02.103679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.103663 5515 manager.go:341] "Starting Device Plugin manager" Oct 30 12:33:02.103757 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:02.103694 5515 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:33:02.103757 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.103705 5515 server.go:87] "Starting device plugin registration server" Oct 30 12:33:02.104087 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.104073 5515 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:33:02.104120 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.104091 5515 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:33:02.104667 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.104195 5515 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:33:02.104821 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.104786 5515 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:33:02.104895 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.104832 5515 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:33:02.104895 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.104840 5515 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:33:02.105007 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.104910 5515 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:33:02.104819127Z","UUID":"0fa3580c-361d-4a4b-8eb9-808d00f74a07","Handler":null,"Name":"","Endpoint":""} Oct 30 12:33:02.106713 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:02.106683 5515 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:33:02.106805 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.106695 5515 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:33:02.106881 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.106826 5515 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:33:02.205183 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.205152 5515 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:33:02.207155 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.207104 5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:33:02.207155 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.207136 5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:33:02.207155 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.207149 5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-193.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:33:02.207295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.207217 5515 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.241160 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.241131 5515 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.241303 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.241291 5515 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.293793 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.293726 5515 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal"] Oct 30 12:33:02.294434 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.294399 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" event={"ID":"f1f1b01344f2c1cc7bc384545bbc0754","Type":"ContainerStarted","Data":"e0993b09ccded105128262e46b28dce042b3dea1f6d1a51b43f39285882cbe87"} Oct 30 12:33:02.294496 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.294443 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" event={"ID":"f1f1b01344f2c1cc7bc384545bbc0754","Type":"ContainerStarted","Data":"8aedbb4e9a0fdbed6117f4cb28a87473625265160767c63a17bab2a3b6b445cf"} Oct 30 12:33:02.294496 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.294477 5515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.294579 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.294480 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" event={"ID":"e707ecff82755bdfb3889f9c2a2af067","Type":"ContainerStarted","Data":"7f0533e363a23595f81c93d3a7670e62f6cfe639dcdf7bae8b1103fe15c23ee6"} Oct 30 12:33:02.294618 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.294578 5515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.294648 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.294612 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" event={"ID":"e707ecff82755bdfb3889f9c2a2af067","Type":"ContainerDied","Data":"b16c01813740e3d093b3d0198934043a5eb0fa8d1b65f621ce1b7b5478b2ba18"} Oct 30 12:33:02.294648 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.294638 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" event={"ID":"e707ecff82755bdfb3889f9c2a2af067","Type":"ContainerStarted","Data":"d149e2fb3ac56044cd991c4a991df3aa3a36e6cda1e1d80f2768fee0ccc96d90"} Oct 30 12:33:02.327965 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.327936 5515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:33:02.328086 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:02.327991 5515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-193.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.329375 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.329364 5515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:33:02.329421 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:02.329388 5515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.407733 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.407698 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1f1b01344f2c1cc7bc384545bbc0754-config\") pod \"kube-apiserver-proxy-ip-10-0-134-193.ec2.internal\" (UID: \"f1f1b01344f2c1cc7bc384545bbc0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.407733 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.407732 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.407905 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.407748 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.508357 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.508259 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1f1b01344f2c1cc7bc384545bbc0754-config\") pod \"kube-apiserver-proxy-ip-10-0-134-193.ec2.internal\" (UID: \"f1f1b01344f2c1cc7bc384545bbc0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.508357 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.508319 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.508357 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.508338 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.508559 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.508378 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.508559 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.508373 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1f1b01344f2c1cc7bc384545bbc0754-config\") pod \"kube-apiserver-proxy-ip-10-0-134-193.ec2.internal\" (UID: \"f1f1b01344f2c1cc7bc384545bbc0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.508559 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.508406 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e707ecff82755bdfb3889f9c2a2af067-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" (UID: \"e707ecff82755bdfb3889f9c2a2af067\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:02.939487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.939445 5515 apiserver.go:52] "Watching apiserver" Oct 30 12:33:02.991240 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.991210 5515 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:33:02.991837 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:02.991811 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal","openshift-multus/multus-additional-cni-plugins-v9vxj","openshift-ovn-kubernetes/ovnkube-node-68xp7","kube-system/konnectivity-agent-d44rw","kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal","openshift-multus/multus-kwjsh","openshift-console/downloads-6dd789c8b7-s2gdv","openshift-ingress-canary/ingress-canary-fqntp","openshift-insights/insights-runtime-extractor-8njh7","openshift-multus/network-metrics-daemon-2j4ks","kube-system/global-pull-secret-syncer-8fl8b","openshift-monitoring/alertmanager-main-0","openshift-monitoring/node-exporter-q2vhb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f","openshift-cluster-node-tuning-operator/tuned-7zvjh","openshift-image-registry/node-ca-wsxsd","openshift-network-operator/iptables-alerter-852jn","openshift-dns/node-resolver-2bkbt","openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs","openshift-dns/dns-default-dxl2t","openshift-monitoring/thanos-querier-55b7b78d57-rlt5q","openshift-network-diagnostics/network-check-target-kjjj6","openshift-console/console-8b6c8b7bd-8d84r"] Oct 30 12:33:03.002281 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.002231 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.008687 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.008660 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5fvrw\"" Oct 30 12:33:03.008687 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.008680 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.008687 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.008680 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.009008 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.008826 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.009008 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.008894 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v9g2h\"" Oct 30 12:33:03.009008 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.008920 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:33:03.009008 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.008945 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.009008 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.008944 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:33:03.009180 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.009130 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:33:03.009180 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.009147 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.009664 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.009651 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:33:03.009858 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.009847 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:33:03.010322 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010306 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:33:03.010437 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010419 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fkzfg\"" Oct 30 12:33:03.010506 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010455 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7nlfh\"" Oct 30 12:33:03.010506 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010469 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sr4bw\"" Oct 30 12:33:03.010506 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010479 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wqghl\"" Oct 30 12:33:03.010677 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010571 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56d7b\"" Oct 30 12:33:03.010677 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010645 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.010813 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010789 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:33:03.010813 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010798 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.010891 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010800 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:33:03.010891 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010833 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.010891 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010852 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:33:03.010891 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.010856 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:33:03.022281 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.022246 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.022447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.022435 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:33:03.024047 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.024032 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.024047 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.024042 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:33:03.024179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.024045 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:33:03.024179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.024095 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 12:33:03.024996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.024983 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.025252 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.025240 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.025330 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.025318 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:33:03.026941 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.026912 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:33:03.026941 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.026921 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:33:03.027075 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.026999 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:33:03.027701 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.027682 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.027784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.027727 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.027784 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.027740 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.027854 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.027789 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 12:33:03.027911 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.027896 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.027942 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.027910 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.028178 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.028167 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:33:03.034522 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.034502 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 12:33:03.034522 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.034515 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 12:33:03.034644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.034528 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5lgc6\"" Oct 30 12:33:03.034644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.034572 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:33:03.035773 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.035759 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-57lkoqn2g03t5\"" Oct 30 12:33:03.035920 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.035903 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zprfc\"" Oct 30 12:33:03.035973 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.035943 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 12:33:03.035973 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.035961 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 12:33:03.036084 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.035973 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 12:33:03.036084 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036053 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 12:33:03.036179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036083 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 12:33:03.036179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036086 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 12:33:03.036179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036108 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kq49t\"" Oct 30 12:33:03.036179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036140 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-47x29\"" Oct 30 12:33:03.036349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036232 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k87jk\"" Oct 30 12:33:03.036349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036231 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-5p94n\"" Oct 30 12:33:03.036349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036297 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7rclv\"" Oct 30 12:33:03.036349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036310 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 12:33:03.036538 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036389 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:33:03.036538 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036490 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.036538 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036502 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 12:33:03.036538 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036525 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-l82fp\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036549 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036554 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-c68n6\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036571 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036583 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036594 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036607 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bklll\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036610 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036640 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036650 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxtxl\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036649 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036679 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036688 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036679 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036715 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 12:33:03.036724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036693 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cgbt6\"" Oct 30 12:33:03.037147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036743 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 12:33:03.037147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036757 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f8qlc\"" Oct 30 12:33:03.037147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036757 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:33:03.037147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036794 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pqqdj\"" Oct 30 12:33:03.037147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.036836 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:33:03.040001 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.039989 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:33:03.040332 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.040319 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 12:33:03.040900 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.040881 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 12:33:03.040999 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.040918 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:33:03.040999 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.040925 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 12:33:03.040999 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.040922 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2gf2w\"" Oct 30 12:33:03.040999 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.040920 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:33:03.055798 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.055777 5515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:03.055922 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.055912 5515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:33:03.062075 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.062054 5515 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:33:03.078924 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.078903 5515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:33:03.079048 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.078922 5515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:33:03.079048 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:03.078949 5515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-193.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-193.ec2.internal" Oct 30 12:33:03.079048 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:03.079007 5515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-193.ec2.internal" Oct 30 12:33:03.111132 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111107 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.111239 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111136 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-tls\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.111239 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111156 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-netns\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.111239 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111171 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24b12750-62c7-478f-bea4-8aee763363f3-hosts-file\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:33:03.111239 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111194 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-systemd\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.111475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111239 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-lib-modules\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.111475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111285 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-netd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.111475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111317 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-config\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.111475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111334 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-registration-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.111475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111363 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.111475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111385 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/475cbe1a-bcd8-4575-9c39-3c3394f062e4-kube-api-access-j8jr9\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:33:03.111475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111420 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-run\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111478 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs5g\" (UniqueName: \"kubernetes.io/projected/0686737a-10d7-4761-877c-46bd3df261c2-kube-api-access-vcs5g\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111504 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-tls\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111510 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f156b-c309-4b4b-8a77-270faeb79648-data-volume\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111534 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-textfile\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111556 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111582 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db5f156b-c309-4b4b-8a77-270faeb79648-data-volume\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111563 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111655 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111663 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-textfile\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111683 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/e3354c62-d061-4f12-9c22-b21455da0faf-kube-api-access-pqzgz\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.111735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111720 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:33:03.112129 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111749 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.112129 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111774 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2e3b448-9d0e-4141-a929-72732655d497-iptables-alerter-script\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:33:03.112129 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111813 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-sys-fs\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.112129 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111842 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-bin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.112129 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111866 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-conf\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.112343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112320 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkcs\" (UniqueName: \"kubernetes.io/projected/24b12750-62c7-478f-bea4-8aee763363f3-kube-api-access-jmkcs\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:33:03.112392 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112367 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.112444 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112397 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:33:03.112444 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112407 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-socket-dir-parent\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.112541 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112446 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-tuned\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.112541 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112464 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-config\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.112541 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112498 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-systemd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.112541 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112529 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-sys\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.112689 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.111934 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.112689 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112541 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-tuned\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.112689 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112610 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f156b-c309-4b4b-8a77-270faeb79648-crio-socket\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.112826 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112757 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2e3b448-9d0e-4141-a929-72732655d497-iptables-alerter-script\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:33:03.112858 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112817 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.112858 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112709 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-system-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.112996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112881 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-daemon-config\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.112996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112929 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-multus-certs\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.112996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.112957 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.113207 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113185 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-daemon-config\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.113258 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113245 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.113342 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113322 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-sys\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.113600 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113577 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.113682 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113627 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.113682 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113665 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.113891 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113874 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.113953 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113919 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.114010 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113951 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhwv\" (UniqueName: \"kubernetes.io/projected/e218ebda-caba-45d2-963a-77b96b0af90f-kube-api-access-7bhwv\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:33:03.114010 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.113983 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpc4l\" (UniqueName: \"kubernetes.io/projected/db5f156b-c309-4b4b-8a77-270faeb79648-kube-api-access-cpc4l\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.114331 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114310 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.114401 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114355 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cnibin\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.114401 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114389 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.114525 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114424 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.114525 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114455 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.114525 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114482 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-tmp\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:33:03.114525 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114514 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.114724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114548 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:33:03.114724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114586 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-konnectivity-ca\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:33:03.114724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114630 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.114724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114659 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc939343-5581-480f-8c67-37ba0d1188c6-config-volume\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:33:03.114724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114690 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2k6\" (UniqueName: \"kubernetes.io/projected/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-kube-api-access-hr2k6\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:33:03.114724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114721 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-kubelet\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114752 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114781 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-kubernetes\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114811 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114841 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3354c62-d061-4f12-9c22-b21455da0faf-ovn-node-metrics-cert\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114868 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-script-lib\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114895 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-root\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114924 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-systemd-units\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114951 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g68t4\" (UniqueName: \"kubernetes.io/projected/f2e3b448-9d0e-4141-a929-72732655d497-kube-api-access-g68t4\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.114979 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-modprobe-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.115034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115007 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-host\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115040 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cni-binary-copy\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115072 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-ovn\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115100 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115124 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115156 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115193 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5ld\" (UniqueName: \"kubernetes.io/projected/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-kube-api-access-rc5ld\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115224 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f156b-c309-4b4b-8a77-270faeb79648-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115253 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115309 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115341 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-etc-kubernetes\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115371 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115403 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc939343-5581-480f-8c67-37ba0d1188c6-tmp-dir\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115431 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbafdcd8-268f-4128-b17e-10d3d92e2227-tmp\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115456 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-system-cni-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115485 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-node-log\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115514 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.115556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115543 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115571 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115602 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-netns\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115632 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-var-lib-kubelet\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115663 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115695 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-log-socket\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115722 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2e3b448-9d0e-4141-a929-72732655d497-host-slash\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115755 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crm9\" (UniqueName: \"kubernetes.io/projected/5f15c7b4-5807-4438-81c7-5a392eca8b2a-kube-api-access-6crm9\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115787 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-var-lib-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115817 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cnibin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115849 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f156b-c309-4b4b-8a77-270faeb79648-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115875 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115907 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7rm\" (UniqueName: \"kubernetes.io/projected/dbafdcd8-268f-4128-b17e-10d3d92e2227-kube-api-access-8r7rm\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115940 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115973 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-etc-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.115999 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116031 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116063 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-tmp\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116104 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdpzv\" (UniqueName: \"kubernetes.io/projected/5b73fd1c-6f57-4172-b98d-c8b8330efd83-kube-api-access-sdpzv\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116143 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116176 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b12750-62c7-478f-bea4-8aee763363f3-tmp-dir\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116215 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-multus\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116246 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116290 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysconfig\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.116471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116318 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5l8w\" (UniqueName: \"kubernetes.io/projected/dc939343-5581-480f-8c67-37ba0d1188c6-kube-api-access-q5l8w\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116654 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116702 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/532d8fee-b407-48ed-b9f5-77b8ad7f7630-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116896 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db5f156b-c309-4b4b-8a77-270faeb79648-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116943 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-wtmp\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.116985 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117130 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-tmp\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117240 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102a0155-f391-4c1e-98d7-eea36a51615e-metrics-certs\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117129 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbafdcd8-268f-4128-b17e-10d3d92e2227-tmp\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117290 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117300 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117310 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117452 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3354c62-d061-4f12-9c22-b21455da0faf-ovn-node-metrics-cert\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117472 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117490 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-socket-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117535 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117576 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.117588 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117588 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.118168 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117750 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-tmp\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.118168 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117747 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.118168 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117792 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db5f156b-c309-4b4b-8a77-270faeb79648-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.118168 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117753 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.118168 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117845 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b12750-62c7-478f-bea4-8aee763363f3-tmp-dir\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:33:03.118168 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.117874 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-konnectivity-ca\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:33:03.118543 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.118518 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.118607 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.118576 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.118679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.118655 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cni-binary-copy\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.118731 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.118713 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.118830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.118806 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.118830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.118830 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.119004 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.118876 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.119004 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.118923 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m58t\" (UniqueName: \"kubernetes.io/projected/fb075adb-3219-4e67-a61a-00a6a422eef7-kube-api-access-5m58t\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.119004 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.118979 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbk6q\" (UniqueName: \"kubernetes.io/projected/102a0155-f391-4c1e-98d7-eea36a51615e-kube-api-access-lbk6q\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:33:03.119159 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119003 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc939343-5581-480f-8c67-37ba0d1188c6-tmp-dir\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:33:03.119159 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119035 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-ovnkube-script-lib\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.119159 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119053 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-thanos-querier-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.119159 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119089 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-hostroot\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.119365 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119168 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc939343-5581-480f-8c67-37ba0d1188c6-config-volume\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:33:03.119365 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119205 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.119365 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119259 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/475cbe1a-bcd8-4575-9c39-3c3394f062e4-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:33:03.119365 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119336 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.119365 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119342 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.119541 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119388 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.119541 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119393 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ecc1b4c7-ef74-4105-b613-2f53486e7aad-original-pull-secret\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:33:03.119541 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119504 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/475cbe1a-bcd8-4575-9c39-3c3394f062e4-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:33:03.119541 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119530 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbafdcd8-268f-4128-b17e-10d3d92e2227-metrics-client-ca\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.119678 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119605 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/532d8fee-b407-48ed-b9f5-77b8ad7f7630-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:33:03.119678 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119638 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-kubelet\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.119678 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119643 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbafdcd8-268f-4128-b17e-10d3d92e2227-metrics-client-ca\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.119678 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119655 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc939343-5581-480f-8c67-37ba0d1188c6-metrics-tls\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:33:03.119809 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119746 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-slash\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.119809 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119760 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/532d8fee-b407-48ed-b9f5-77b8ad7f7630-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:33:03.119892 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119849 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-env-overrides\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.119892 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119874 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc939343-5581-480f-8c67-37ba0d1188c6-metrics-tls\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:33:03.119892 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119880 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.119985 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.119928 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddf98\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.120091 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120062 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3354c62-d061-4f12-9c22-b21455da0faf-env-overrides\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.120146 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120103 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.120146 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120077 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e218ebda-caba-45d2-963a-77b96b0af90f-serviceca\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:33:03.120210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120183 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.120240 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120213 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2b7j\" (UniqueName: \"kubernetes.io/projected/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-kube-api-access-d2b7j\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.120297 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120241 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.120297 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120242 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.120372 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120310 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-bin\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.120421 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120407 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.120458 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120429 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-device-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.120492 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120463 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhkph\" (UniqueName: \"kubernetes.io/projected/532d8fee-b407-48ed-b9f5-77b8ad7f7630-kube-api-access-nhkph\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:33:03.120528 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120506 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-k8s-cni-cncf-io\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.120563 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120534 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-grpc-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.120597 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120564 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-kubelet-config\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:33:03.120597 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120586 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0686737a-10d7-4761-877c-46bd3df261c2-cert\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:33:03.120683 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120601 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-conf-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.120683 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120616 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e218ebda-caba-45d2-963a-77b96b0af90f-host\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:33:03.120683 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120669 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-dbus\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:33:03.120830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120696 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-agent-certs\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:33:03.120830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120718 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-metrics-client-ca\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.120830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120745 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e218ebda-caba-45d2-963a-77b96b0af90f-serviceca\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:33:03.120830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120769 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbafdcd8-268f-4128-b17e-10d3d92e2227-secret-grpc-tls\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.120967 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120834 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0686737a-10d7-4761-877c-46bd3df261c2-cert\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:33:03.120967 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120852 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b73fd1c-6f57-4172-b98d-c8b8330efd83-metrics-client-ca\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.120967 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.120956 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b35dcc8-6a2f-4cc9-a810-85ec056b5899-agent-certs\") pod \"konnectivity-agent-d44rw\" (UID: \"5b35dcc8-6a2f-4cc9-a810-85ec056b5899\") " pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:33:03.134766 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.134732 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/e3354c62-d061-4f12-9c22-b21455da0faf-kube-api-access-pqzgz\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.134766 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.134738 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkcs\" (UniqueName: \"kubernetes.io/projected/24b12750-62c7-478f-bea4-8aee763363f3-kube-api-access-jmkcs\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:33:03.134970 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.134948 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhwv\" (UniqueName: \"kubernetes.io/projected/e218ebda-caba-45d2-963a-77b96b0af90f-kube-api-access-7bhwv\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:33:03.135003 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.134956 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/475cbe1a-bcd8-4575-9c39-3c3394f062e4-kube-api-access-j8jr9\") pod \"monitoring-plugin-77f4f76594-q6hgs\" (UID: \"475cbe1a-bcd8-4575-9c39-3c3394f062e4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:33:03.135148 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.135135 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs5g\" (UniqueName: \"kubernetes.io/projected/0686737a-10d7-4761-877c-46bd3df261c2-kube-api-access-vcs5g\") pod \"ingress-canary-fqntp\" (UID: \"0686737a-10d7-4761-877c-46bd3df261c2\") " pod="openshift-ingress-canary/ingress-canary-fqntp" Oct 30 12:33:03.137690 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.137674 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c\") pod \"console-8b6c8b7bd-8d84r\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.142017 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.142001 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5l8w\" (UniqueName: \"kubernetes.io/projected/dc939343-5581-480f-8c67-37ba0d1188c6-kube-api-access-q5l8w\") pod \"dns-default-dxl2t\" (UID: \"dc939343-5581-480f-8c67-37ba0d1188c6\") " pod="openshift-dns/dns-default-dxl2t" Oct 30 12:33:03.142068 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.142047 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpc4l\" (UniqueName: \"kubernetes.io/projected/db5f156b-c309-4b4b-8a77-270faeb79648-kube-api-access-cpc4l\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.142098 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.142072 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5zc\" (UniqueName: \"kubernetes.io/projected/88627b5e-75be-4029-ba55-51759aedc92d-kube-api-access-bt5zc\") pod \"network-check-target-kjjj6\" (UID: \"88627b5e-75be-4029-ba55-51759aedc92d\") " pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:33:03.145770 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.145740 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2b7j\" (UniqueName: \"kubernetes.io/projected/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-kube-api-access-d2b7j\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.146248 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.146229 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5ld\" (UniqueName: \"kubernetes.io/projected/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-kube-api-access-rc5ld\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.146441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.146426 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdpzv\" (UniqueName: \"kubernetes.io/projected/5b73fd1c-6f57-4172-b98d-c8b8330efd83-kube-api-access-sdpzv\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.146674 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.146647 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g68t4\" (UniqueName: \"kubernetes.io/projected/f2e3b448-9d0e-4141-a929-72732655d497-kube-api-access-g68t4\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:33:03.146717 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.146704 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhkph\" (UniqueName: \"kubernetes.io/projected/532d8fee-b407-48ed-b9f5-77b8ad7f7630-kube-api-access-nhkph\") pod \"prometheus-operator-9cd89b74c-lvsdk\" (UID: \"532d8fee-b407-48ed-b9f5-77b8ad7f7630\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lvsdk" Oct 30 12:33:03.147061 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.147041 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m58t\" (UniqueName: \"kubernetes.io/projected/fb075adb-3219-4e67-a61a-00a6a422eef7-kube-api-access-5m58t\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.147433 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.147418 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7rm\" (UniqueName: \"kubernetes.io/projected/dbafdcd8-268f-4128-b17e-10d3d92e2227-kube-api-access-8r7rm\") pod \"thanos-querier-55b7b78d57-rlt5q\" (UID: \"dbafdcd8-268f-4128-b17e-10d3d92e2227\") " pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:03.147529 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.147515 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2k6\" (UniqueName: \"kubernetes.io/projected/983c8f5c-68cf-42a4-b931-4ceb2b7e1e77-kube-api-access-hr2k6\") pod \"downloads-6dd789c8b7-s2gdv\" (UID: \"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77\") " pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:33:03.148591 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.148575 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbk6q\" (UniqueName: \"kubernetes.io/projected/102a0155-f391-4c1e-98d7-eea36a51615e-kube-api-access-lbk6q\") pod \"network-metrics-daemon-2j4ks\" (UID: \"102a0155-f391-4c1e-98d7-eea36a51615e\") " pod="openshift-multus/network-metrics-daemon-2j4ks" Oct 30 12:33:03.150587 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.150571 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddf98\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98\") pod \"alertmanager-main-0\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:03.151996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.151981 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crm9\" (UniqueName: \"kubernetes.io/projected/5f15c7b4-5807-4438-81c7-5a392eca8b2a-kube-api-access-6crm9\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.221937 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.221861 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.221937 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.221898 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-sys-fs\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.221937 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.221914 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-bin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.221937 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.221930 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-conf\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.221952 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.221973 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-socket-dir-parent\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.221999 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-systemd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222004 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-sys-fs\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222011 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-bin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222022 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-sys\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.221997 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222037 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222050 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-systemd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222062 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-socket-dir-parent\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222046 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f156b-c309-4b4b-8a77-270faeb79648-crio-socket\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222086 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-sys\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222103 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db5f156b-c309-4b4b-8a77-270faeb79648-crio-socket\") pod \"insights-runtime-extractor-8njh7\" (UID: \"db5f156b-c309-4b4b-8a77-270faeb79648\") " pod="openshift-insights/insights-runtime-extractor-8njh7" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222107 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-conf\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222111 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-system-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222143 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-system-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222159 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-multus-certs\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222186 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-multus-certs\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222188 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-sys\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.222210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222216 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cnibin\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222242 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222277 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-sys\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222301 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-kubelet\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222327 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-kubernetes\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222351 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-kubelet\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222351 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-root\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222303 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-cnibin\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222376 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-systemd-units\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222391 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-kubernetes\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222398 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222444 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-root\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222456 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-modprobe-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222479 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-systemd-units\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222512 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-host\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222547 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-host\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222554 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-ovn\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222573 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-modprobe-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222613 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-etc-kubernetes\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222634 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-ovn\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222643 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-etc-kubernetes\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222648 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-system-cni-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222672 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-node-log\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222696 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222720 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-netns\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222725 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f15c7b4-5807-4438-81c7-5a392eca8b2a-system-cni-dir\") pod \"multus-additional-cni-plugins-v9vxj\" (UID: \"5f15c7b4-5807-4438-81c7-5a392eca8b2a\") " pod="openshift-multus/multus-additional-cni-plugins-v9vxj" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222728 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-node-log\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222744 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-ovn-kubernetes\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222741 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-var-lib-kubelet\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222757 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-netns\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222772 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-log-socket\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222792 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2e3b448-9d0e-4141-a929-72732655d497-host-slash\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222793 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-var-lib-kubelet\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222814 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-log-socket\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222815 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-var-lib-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222834 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2e3b448-9d0e-4141-a929-72732655d497-host-slash\") pod \"iptables-alerter-852jn\" (UID: \"f2e3b448-9d0e-4141-a929-72732655d497\") " pod="openshift-network-operator/iptables-alerter-852jn" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222842 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-var-lib-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222849 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cnibin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222879 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-etc-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222905 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222933 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-multus\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222942 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-cnibin\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222959 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-run-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222961 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-etc-openvswitch\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222958 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysconfig\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222978 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-cni-multus\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.222994 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-wtmp\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223015 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-socket-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223022 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysconfig\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223041 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-hostroot\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223060 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b73fd1c-6f57-4172-b98d-c8b8330efd83-node-exporter-wtmp\") pod \"node-exporter-q2vhb\" (UID: \"5b73fd1c-6f57-4172-b98d-c8b8330efd83\") " pod="openshift-monitoring/node-exporter-q2vhb" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223066 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223078 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-socket-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223090 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223099 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-hostroot\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223121 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-kubelet\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223133 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-cni-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223142 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-slash\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223168 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-bin\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223175 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-var-lib-kubelet\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223190 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-device-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223196 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-slash\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223200 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-sysctl-d\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223213 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-bin\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223214 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-k8s-cni-cncf-io\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223236 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-device-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223237 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-host-run-k8s-cni-cncf-io\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223247 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-kubelet-config\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:33:03.223224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223290 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-conf-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223310 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-kubelet-config\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223312 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e218ebda-caba-45d2-963a-77b96b0af90f-host\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223335 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-dbus\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223339 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e218ebda-caba-45d2-963a-77b96b0af90f-host\") pod \"node-ca-wsxsd\" (UID: \"e218ebda-caba-45d2-963a-77b96b0af90f\") " pod="openshift-image-registry/node-ca-wsxsd" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223352 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-netns\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223363 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38-multus-conf-dir\") pod \"multus-kwjsh\" (UID: \"10a5fae8-0a72-4dcd-99b7-4cc8c7c30c38\") " pod="openshift-multus/multus-kwjsh" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223375 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24b12750-62c7-478f-bea4-8aee763363f3-hosts-file\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223397 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-run-netns\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223405 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ecc1b4c7-ef74-4105-b613-2f53486e7aad-dbus\") pod \"global-pull-secret-syncer-8fl8b\" (UID: \"ecc1b4c7-ef74-4105-b613-2f53486e7aad\") " pod="kube-system/global-pull-secret-syncer-8fl8b" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223422 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24b12750-62c7-478f-bea4-8aee763363f3-hosts-file\") pod \"node-resolver-2bkbt\" (UID: \"24b12750-62c7-478f-bea4-8aee763363f3\") " pod="openshift-dns/node-resolver-2bkbt" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223427 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-systemd\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223457 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-lib-modules\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223462 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-etc-systemd\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223483 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-netd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223507 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-registration-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223522 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-lib-modules\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223530 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223556 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-run\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223583 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-registration-dir\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223595 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb075adb-3219-4e67-a61a-00a6a422eef7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p2g9f\" (UID: \"fb075adb-3219-4e67-a61a-00a6a422eef7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p2g9f" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223596 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3354c62-d061-4f12-9c22-b21455da0faf-host-cni-netd\") pod \"ovnkube-node-68xp7\" (UID: \"e3354c62-d061-4f12-9c22-b21455da0faf\") " pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:03.225177 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.223627 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0-run\") pod \"tuned-7zvjh\" (UID: \"e46c98aa-50ec-4a0f-b2e9-f89f777a1bc0\") " pod="openshift-cluster-node-tuning-operator/tuned-7zvjh" Oct 30 12:33:03.302093 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.302072 5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:33:03.302780 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.302752 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:03.595220 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:03.595189 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b6c8b7bd-8d84r"] Oct 30 12:33:03.597937 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:03.597907 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8c4da3_8154_4b51_ac73_49afe96db85e.slice/crio-b08e46fbd7ed086401c0c4cc66fd127b646cbc6b3eb75130d8d7393753eeb95c WatchSource:0}: Error finding container b08e46fbd7ed086401c0c4cc66fd127b646cbc6b3eb75130d8d7393753eeb95c: Status 404 returned error can't find the container with id b08e46fbd7ed086401c0c4cc66fd127b646cbc6b3eb75130d8d7393753eeb95c Oct 30 12:33:04.061088 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:04.061048 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b6c8b7bd-8d84r" event={"ID":"8a8c4da3-8154-4b51-ac73-49afe96db85e","Type":"ContainerStarted","Data":"b08e46fbd7ed086401c0c4cc66fd127b646cbc6b3eb75130d8d7393753eeb95c"} Oct 30 12:33:04.108196 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:04.108161 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-q6hgs" Oct 30 12:33:04.346310 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:04.346218 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:04.461255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:04.461227 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dxl2t" Oct 30 12:33:04.542023 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:04.541942 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:05.078201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:05.077297 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" Oct 30 12:33:05.345557 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:05.345477 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:05.465362 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:05.465241 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:05.697256 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:05.697217 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:07.074613 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:07.074534 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b6c8b7bd-8d84r" event={"ID":"8a8c4da3-8154-4b51-ac73-49afe96db85e","Type":"ContainerStarted","Data":"e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a"} Oct 30 12:33:09.712490 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:09.712450 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:33:11.754927 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:11.754895 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:11.755346 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:11.755117 5515 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:33:12.496226 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:12.496182 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:12.715623 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:12.715580 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:12.884457 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:12.884416 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:13.198501 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.198415 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f79bf4f6d-b29lv"] Oct 30 12:33:13.201857 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.201834 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.253579 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.253548 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 12:33:13.254241 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.254217 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f79bf4f6d-b29lv"] Oct 30 12:33:13.320162 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.320126 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hqn\" (UniqueName: \"kubernetes.io/projected/5a504de6-2016-4d68-9602-dc0ff0c2d192-kube-api-access-x9hqn\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.320375 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.320172 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-trusted-ca-bundle\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.320375 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.320299 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-serving-cert\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.320375 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.320349 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-config\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.320533 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.320391 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a504de6-2016-4d68-9602-dc0ff0c2d192-tmp\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.320533 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.320509 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-oauth-serving-cert\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.320620 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.320555 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-service-ca\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.320620 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.320580 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-oauth-config\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.421285 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.421218 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-config\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.421477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.421296 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a504de6-2016-4d68-9602-dc0ff0c2d192-tmp\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.421477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.421328 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-oauth-serving-cert\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.421477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.421352 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-service-ca\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.421477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.421377 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-oauth-config\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.421477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.421418 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hqn\" (UniqueName: \"kubernetes.io/projected/5a504de6-2016-4d68-9602-dc0ff0c2d192-kube-api-access-x9hqn\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.421477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.421442 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-trusted-ca-bundle\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.421477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.421479 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-serving-cert\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.422086 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.421910 5515 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:33:13.422301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.422167 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a504de6-2016-4d68-9602-dc0ff0c2d192-tmp\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.422666 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.422642 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-config\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.422768 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.422642 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-oauth-serving-cert\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.422967 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.422947 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-trusted-ca-bundle\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.423037 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.422975 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-service-ca\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.424496 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.424476 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-oauth-config\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.424585 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.424559 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-serving-cert\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.445581 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.445556 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hqn\" (UniqueName: \"kubernetes.io/projected/5a504de6-2016-4d68-9602-dc0ff0c2d192-kube-api-access-x9hqn\") pod \"console-7f79bf4f6d-b29lv\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.514260 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.514175 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:13.816371 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:13.816289 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f79bf4f6d-b29lv"] Oct 30 12:33:13.820338 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:13.820306 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a504de6_2016_4d68_9602_dc0ff0c2d192.slice/crio-597a5357b2a718584e3c3955f7d6804fb8ce4119c739edd0320b849db0074e85 WatchSource:0}: Error finding container 597a5357b2a718584e3c3955f7d6804fb8ce4119c739edd0320b849db0074e85: Status 404 returned error can't find the container with id 597a5357b2a718584e3c3955f7d6804fb8ce4119c739edd0320b849db0074e85 Oct 30 12:33:14.102969 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:14.102879 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f79bf4f6d-b29lv" event={"ID":"5a504de6-2016-4d68-9602-dc0ff0c2d192","Type":"ContainerStarted","Data":"69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2"} Oct 30 12:33:14.102969 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:14.102941 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f79bf4f6d-b29lv" event={"ID":"5a504de6-2016-4d68-9602-dc0ff0c2d192","Type":"ContainerStarted","Data":"597a5357b2a718584e3c3955f7d6804fb8ce4119c739edd0320b849db0074e85"} Oct 30 12:33:14.146465 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:14.146411 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f79bf4f6d-b29lv" podStartSLOduration=1.146398192 podStartE2EDuration="1.146398192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:14.146170142 +0000 UTC m=+12.272971040" watchObservedRunningTime="2026-10-30 12:33:14.146398192 +0000 UTC m=+12.273199082" Oct 30 12:33:15.785172 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:15.785135 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:15.790499 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:15.790475 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:16.115259 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:16.115175 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:17.486173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:17.486133 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:17.686890 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:17.686851 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:19.124164 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:19.124127 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-s2gdv" event={"ID":"983c8f5c-68cf-42a4-b931-4ceb2b7e1e77","Type":"ContainerStarted","Data":"46b8a598e078270a09919cbed28bbf4f517b438c5b8e12b7cd4a9d54735dd1a6"} Oct 30 12:33:19.124607 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:19.124429 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:33:19.126042 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:19.126013 5515 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-s2gdv container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.13:8080/\": dial tcp 10.134.0.13:8080: connect: connection refused" start-of-body= Oct 30 12:33:19.126199 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:19.126081 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-s2gdv" podUID="983c8f5c-68cf-42a4-b931-4ceb2b7e1e77" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.13:8080/\": dial tcp 10.134.0.13:8080: connect: connection refused" Oct 30 12:33:20.146528 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:20.146491 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-s2gdv" Oct 30 12:33:20.364892 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:20.364853 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-68xp7" Oct 30 12:33:22.223591 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:22.223516 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:22.477645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:22.477552 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:23.515171 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:23.515131 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:23.515663 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:23.515221 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:23.520990 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:23.520952 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:24.147468 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:24.147433 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:33:24.228227 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:24.228194 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b6c8b7bd-8d84r"] Oct 30 12:33:26.087325 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:26.087285 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:26.291367 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:26.291322 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:27.973079 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:27.973054 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fqntp_0686737a-10d7-4761-877c-46bd3df261c2/serve-healthcheck-canary/0.log" Oct 30 12:33:28.284566 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:28.284479 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:28.502562 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:28.502523 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:28.986344 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:28.986308 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:33:29.519606 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:29.519574 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:33:29.519911 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:29.519888 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="alertmanager" containerID="cri-o://63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac" gracePeriod=120 Oct 30 12:33:29.520001 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:29.519967 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy-metric" containerID="cri-o://c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39" gracePeriod=120 Oct 30 12:33:29.520255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:29.519983 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="prom-label-proxy" containerID="cri-o://1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48" gracePeriod=120 Oct 30 12:33:29.520370 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:29.520003 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="config-reloader" containerID="cri-o://498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846" gracePeriod=120 Oct 30 12:33:29.520370 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:29.520013 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy" containerID="cri-o://7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4" gracePeriod=120 Oct 30 12:33:29.520459 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:29.520011 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy-web" containerID="cri-o://400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4" gracePeriod=120 Oct 30 12:33:30.172238 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172203 5515 generic.go:356] "Generic (PLEG): container finished" podID="4ea59390-80c3-432c-bda5-88af53de9263" containerID="1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48" exitCode=0 Oct 30 12:33:30.172238 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172230 5515 generic.go:356] "Generic (PLEG): container finished" podID="4ea59390-80c3-432c-bda5-88af53de9263" containerID="c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39" exitCode=0 Oct 30 12:33:30.172238 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172236 5515 generic.go:356] "Generic (PLEG): container finished" podID="4ea59390-80c3-432c-bda5-88af53de9263" containerID="7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4" exitCode=0 Oct 30 12:33:30.172238 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172244 5515 generic.go:356] "Generic (PLEG): container finished" podID="4ea59390-80c3-432c-bda5-88af53de9263" containerID="498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846" exitCode=0 Oct 30 12:33:30.172740 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172255 5515 generic.go:356] "Generic (PLEG): container finished" podID="4ea59390-80c3-432c-bda5-88af53de9263" containerID="63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac" exitCode=0 Oct 30 12:33:30.172740 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172294 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerDied","Data":"1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48"} Oct 30 12:33:30.172740 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172370 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerDied","Data":"c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39"} Oct 30 12:33:30.172740 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172397 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerDied","Data":"7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4"} Oct 30 12:33:30.172740 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172421 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerDied","Data":"498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846"} Oct 30 12:33:30.172740 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.172443 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerDied","Data":"63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac"} Oct 30 12:33:30.703370 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.700958 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:30.791427 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.791388 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:30.873152 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873071 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873152 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873115 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873152 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873140 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873170 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873192 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873208 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873246 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873302 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873364 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873425 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873461 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873501 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873541 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98\") pod \"4ea59390-80c3-432c-bda5-88af53de9263\" (UID: \"4ea59390-80c3-432c-bda5-88af53de9263\") " Oct 30 12:33:30.873810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873689 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:30.874043 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873831 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-metrics-client-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.874043 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.873933 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:30.874890 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.874551 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:30.877303 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.877228 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:30.877585 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.877558 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:30.877688 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.877640 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:30.877731 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.877700 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:30.877826 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.877797 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "kube-api-access-ddf98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:30.877918 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.877825 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:30.878003 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.877979 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:30.878106 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.878086 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:30.888485 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.885192 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:30.890571 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.890536 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config" pod "4ea59390-80c3-432c-bda5-88af53de9263" (UID: "4ea59390-80c3-432c-bda5-88af53de9263"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:30.923954 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.923917 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:30.975124 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975084 5515 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975124 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975117 5515 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975124 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975130 5515 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-config-out\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975141 5515 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-main-db\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975150 5515 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975158 5515 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-tls-assets\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975166 5515 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-web-config\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975174 5515 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975182 5515 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-cluster-tls-config\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975191 5515 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4ea59390-80c3-432c-bda5-88af53de9263-secret-alertmanager-main-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975200 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddf98\" (UniqueName: \"kubernetes.io/projected/4ea59390-80c3-432c-bda5-88af53de9263-kube-api-access-ddf98\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.975378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:30.975209 5515 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea59390-80c3-432c-bda5-88af53de9263-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:31.179462 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.179425 5515 generic.go:356] "Generic (PLEG): container finished" podID="4ea59390-80c3-432c-bda5-88af53de9263" containerID="400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4" exitCode=0 Oct 30 12:33:31.179850 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.179490 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerDied","Data":"400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4"} Oct 30 12:33:31.179850 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.179528 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.179850 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.179555 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4ea59390-80c3-432c-bda5-88af53de9263","Type":"ContainerDied","Data":"6d965288c14cb8d369763f0f7b69aa1295310a7bcc9177dd32191a483e8987f9"} Oct 30 12:33:31.179850 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.179600 5515 scope.go:122] "RemoveContainer" containerID="1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48" Oct 30 12:33:31.187258 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.187241 5515 scope.go:122] "RemoveContainer" containerID="c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39" Oct 30 12:33:31.193738 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.193717 5515 scope.go:122] "RemoveContainer" containerID="7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4" Oct 30 12:33:31.199964 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.199948 5515 scope.go:122] "RemoveContainer" containerID="400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4" Oct 30 12:33:31.206030 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.206017 5515 scope.go:122] "RemoveContainer" containerID="498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846" Oct 30 12:33:31.212451 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.212434 5515 scope.go:122] "RemoveContainer" containerID="63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac" Oct 30 12:33:31.219192 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.219173 5515 scope.go:122] "RemoveContainer" containerID="4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024" Oct 30 12:33:31.226004 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.225987 5515 scope.go:122] "RemoveContainer" containerID="1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48" Oct 30 12:33:31.226310 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:31.226283 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48\": container with ID starting with 1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48 not found: ID does not exist" containerID="1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48" Oct 30 12:33:31.226394 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.226321 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48"} err="failed to get container status \"1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48\": rpc error: code = NotFound desc = could not find container \"1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48\": container with ID starting with 1d449021cf5717427564feb103325fba1bd92358354a63639e644668ae1b0e48 not found: ID does not exist" Oct 30 12:33:31.226394 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.226367 5515 scope.go:122] "RemoveContainer" containerID="c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39" Oct 30 12:33:31.226636 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:31.226618 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39\": container with ID starting with c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39 not found: ID does not exist" containerID="c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39" Oct 30 12:33:31.226680 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.226641 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39"} err="failed to get container status \"c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39\": rpc error: code = NotFound desc = could not find container \"c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39\": container with ID starting with c4daa06b305da7ee30db4a53223f280d4e8e22e8617cde207e9dc8f9b6f21f39 not found: ID does not exist" Oct 30 12:33:31.226680 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.226654 5515 scope.go:122] "RemoveContainer" containerID="7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4" Oct 30 12:33:31.226851 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:31.226836 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4\": container with ID starting with 7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4 not found: ID does not exist" containerID="7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4" Oct 30 12:33:31.226893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.226854 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4"} err="failed to get container status \"7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4\": rpc error: code = NotFound desc = could not find container \"7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4\": container with ID starting with 7ef589fd603d2675fb70b764d8171b95cd618578883dba46c1f9082891945de4 not found: ID does not exist" Oct 30 12:33:31.226893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.226866 5515 scope.go:122] "RemoveContainer" containerID="400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4" Oct 30 12:33:31.227050 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:31.227035 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4\": container with ID starting with 400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4 not found: ID does not exist" containerID="400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4" Oct 30 12:33:31.227084 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.227053 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4"} err="failed to get container status \"400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4\": rpc error: code = NotFound desc = could not find container \"400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4\": container with ID starting with 400cf23639206ede4d01ec7d511dd3ca4fdf6402d11b925dd9e54c9d6e430ba4 not found: ID does not exist" Oct 30 12:33:31.227084 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.227064 5515 scope.go:122] "RemoveContainer" containerID="498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846" Oct 30 12:33:31.227255 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:31.227240 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846\": container with ID starting with 498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846 not found: ID does not exist" containerID="498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846" Oct 30 12:33:31.227419 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.227260 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846"} err="failed to get container status \"498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846\": rpc error: code = NotFound desc = could not find container \"498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846\": container with ID starting with 498d8dd24b7f3f199a05d70d55a5d2ce079d1b4f4e0a38e6e1c82ecdbfa6c846 not found: ID does not exist" Oct 30 12:33:31.227419 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.227286 5515 scope.go:122] "RemoveContainer" containerID="63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac" Oct 30 12:33:31.227509 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:31.227495 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac\": container with ID starting with 63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac not found: ID does not exist" containerID="63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac" Oct 30 12:33:31.227543 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.227510 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac"} err="failed to get container status \"63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac\": rpc error: code = NotFound desc = could not find container \"63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac\": container with ID starting with 63f495bed1fed28dd70181f6f775f2db929dffaee73dd5fa684da2a5a06674ac not found: ID does not exist" Oct 30 12:33:31.227543 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.227520 5515 scope.go:122] "RemoveContainer" containerID="4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024" Oct 30 12:33:31.227695 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:31.227680 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024\": container with ID starting with 4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024 not found: ID does not exist" containerID="4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024" Oct 30 12:33:31.227733 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.227698 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024"} err="failed to get container status \"4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024\": rpc error: code = NotFound desc = could not find container \"4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024\": container with ID starting with 4c43c15dcf08922301b5ef0a927efecb28d5ce858a5a01bc6a9bd9fdea77f024 not found: ID does not exist" Oct 30 12:33:31.231694 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.231673 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:33:31.243595 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.243574 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:33:31.277042 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.276919 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:33:31.277264 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277249 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="config-reloader" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277281 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="config-reloader" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277298 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="prom-label-proxy" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277303 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="prom-label-proxy" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277313 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277318 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277328 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="init-config-reloader" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277332 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="init-config-reloader" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277341 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy-web" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277346 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy-web" Oct 30 12:33:31.277352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277356 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="alertmanager" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277361 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="alertmanager" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277371 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy-metric" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277375 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy-metric" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277421 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="alertmanager" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277428 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy-metric" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277436 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="config-reloader" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277443 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="prom-label-proxy" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277451 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277457 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="init-config-reloader" Oct 30 12:33:31.277641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.277462 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea59390-80c3-432c-bda5-88af53de9263" containerName="kube-rbac-proxy-web" Oct 30 12:33:31.305032 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.304995 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.310252 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.310223 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:33:31.320008 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.319985 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 12:33:31.320293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.320262 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 12:33:31.320371 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.320320 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 12:33:31.320371 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.320322 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 12:33:31.320489 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.320474 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 12:33:31.320593 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.320580 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pqqdj\"" Oct 30 12:33:31.320646 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.320596 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 12:33:31.320646 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.320584 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 12:33:31.321467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.321452 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 12:33:31.332523 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.332501 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 12:33:31.378238 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378203 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378418 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378321 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378418 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378353 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rsh7\" (UniqueName: \"kubernetes.io/projected/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-kube-api-access-6rsh7\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378513 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378426 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378549 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378507 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378599 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378580 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378636 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378618 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378638 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-config-out\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378670 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378751 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378730 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378785 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378773 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-web-config\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378824 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378793 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.378863 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.378819 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480194 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480240 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480523 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480349 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480523 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480393 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-config-out\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480523 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480416 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480523 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480455 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480523 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480492 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-web-config\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480594 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480640 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480686 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480717 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480742 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsh7\" (UniqueName: \"kubernetes.io/projected/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-kube-api-access-6rsh7\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.480776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.480776 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.482206 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.481557 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.482206 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.481678 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.482725 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.482682 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.483380 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.483199 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-config-out\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.483503 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.483482 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.483589 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.483533 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-web-config\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.483675 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.483646 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.483941 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.483915 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.484024 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.484003 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.484348 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.484331 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.484694 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.484679 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.484838 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.484818 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.504607 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.504588 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rsh7\" (UniqueName: \"kubernetes.io/projected/f7dc73ed-382c-4a8e-a1e1-7621edf6cb79-kube-api-access-6rsh7\") pod \"alertmanager-main-0\" (UID: \"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.716175 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.716143 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:33:31.999318 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:31.999234 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea59390-80c3-432c-bda5-88af53de9263" path="/var/lib/kubelet/pods/4ea59390-80c3-432c-bda5-88af53de9263/volumes" Oct 30 12:33:32.060936 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:32.060894 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:33:32.186428 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:32.186389 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79","Type":"ContainerStarted","Data":"47d361375d5abc912df7ea6eb539e13776028e3ff606440aa0ce22d6bd5bdf0a"} Oct 30 12:33:32.186800 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:32.186475 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79","Type":"ContainerStarted","Data":"7918fb6822eae61e684ff4b620454480076cdb682fe84e0a6c122235e9c36730"} Oct 30 12:33:32.834065 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:32.834035 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:33:33.192347 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.192313 5515 generic.go:356] "Generic (PLEG): container finished" podID="f7dc73ed-382c-4a8e-a1e1-7621edf6cb79" containerID="47d361375d5abc912df7ea6eb539e13776028e3ff606440aa0ce22d6bd5bdf0a" exitCode=0 Oct 30 12:33:33.192730 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.192352 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79","Type":"ContainerDied","Data":"47d361375d5abc912df7ea6eb539e13776028e3ff606440aa0ce22d6bd5bdf0a"} Oct 30 12:33:33.469330 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.469301 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-874fd879-b99jw"] Oct 30 12:33:33.503560 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.503529 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-874fd879-b99jw"] Oct 30 12:33:33.503765 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.503652 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.518913 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.518794 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 12:33:33.518913 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.518859 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vlwwx\"" Oct 30 12:33:33.518913 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.518879 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 12:33:33.518913 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.518874 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 12:33:33.518913 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.518919 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 12:33:33.582670 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.582645 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 12:33:33.595417 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.595388 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 12:33:33.596500 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.596478 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.596563 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.596511 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqfn5\" (UniqueName: \"kubernetes.io/projected/98b4567f-f0d4-47a3-a8a3-3cac381e107d-kube-api-access-tqfn5\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.596563 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.596544 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-telemeter-client-tls\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.596647 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.596630 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-secret-telemeter-client\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.596679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.596660 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b4567f-f0d4-47a3-a8a3-3cac381e107d-serving-certs-ca-bundle\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.596711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.596680 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b4567f-f0d4-47a3-a8a3-3cac381e107d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.596778 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.596766 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98b4567f-f0d4-47a3-a8a3-3cac381e107d-metrics-client-ca\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.596818 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.596794 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-federate-client-tls\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.697397 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.697374 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfn5\" (UniqueName: \"kubernetes.io/projected/98b4567f-f0d4-47a3-a8a3-3cac381e107d-kube-api-access-tqfn5\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.697509 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.697414 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-telemeter-client-tls\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.697509 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.697448 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-secret-telemeter-client\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.697509 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.697478 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b4567f-f0d4-47a3-a8a3-3cac381e107d-serving-certs-ca-bundle\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.697509 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.697503 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b4567f-f0d4-47a3-a8a3-3cac381e107d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.697699 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.697539 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98b4567f-f0d4-47a3-a8a3-3cac381e107d-metrics-client-ca\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.697699 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.697583 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-federate-client-tls\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.697699 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.697641 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.698531 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.698434 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98b4567f-f0d4-47a3-a8a3-3cac381e107d-metrics-client-ca\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.698531 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.698454 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b4567f-f0d4-47a3-a8a3-3cac381e107d-serving-certs-ca-bundle\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.698531 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.698472 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b4567f-f0d4-47a3-a8a3-3cac381e107d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.700020 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.699992 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-secret-telemeter-client\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.700150 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.700121 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-federate-client-tls\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.700217 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.700203 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.700278 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.700223 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/98b4567f-f0d4-47a3-a8a3-3cac381e107d-telemeter-client-tls\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.721355 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.720479 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfn5\" (UniqueName: \"kubernetes.io/projected/98b4567f-f0d4-47a3-a8a3-3cac381e107d-kube-api-access-tqfn5\") pod \"telemeter-client-874fd879-b99jw\" (UID: \"98b4567f-f0d4-47a3-a8a3-3cac381e107d\") " pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:33.877101 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:33.876875 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-874fd879-b99jw" Oct 30 12:33:34.203485 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:34.203454 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79","Type":"ContainerStarted","Data":"b713da1b73688f89585a83a42d0d81a7cfcf6b1263c84846274de70e67fa2d22"} Oct 30 12:33:34.203893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:34.203520 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79","Type":"ContainerStarted","Data":"82db949e79e282a808e988457ee80b798c68fd28a40fc18d5a46fe662ddb3316"} Oct 30 12:33:34.203893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:34.203548 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79","Type":"ContainerStarted","Data":"3beaa8d65dc37a50bcf5dddbe57017cf32aa6d97c1346a73d673788256035e14"} Oct 30 12:33:34.203893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:34.203573 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79","Type":"ContainerStarted","Data":"8ac870a8571bdbae964f4f4a31ee9f9eef87b73b0ae4da85a249de4327f558fe"} Oct 30 12:33:34.203893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:34.203595 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79","Type":"ContainerStarted","Data":"2787ee39edded15f6c2e02c6af3612a528b0fd100e2dd95d45b3105ab9fb0439"} Oct 30 12:33:34.203893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:34.203617 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7dc73ed-382c-4a8e-a1e1-7621edf6cb79","Type":"ContainerStarted","Data":"a4b9cf0ab16dec6ce5b9878329cd036488079bfa97a17ad167c5ea1ad1e2737a"} Oct 30 12:33:34.213943 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:34.213919 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-874fd879-b99jw"] Oct 30 12:33:34.215775 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:34.215747 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b4567f_f0d4_47a3_a8a3_3cac381e107d.slice/crio-64a79a4183033bd4e9118c69d6ab531713c7212a93d50fedd1970ba439c04a00 WatchSource:0}: Error finding container 64a79a4183033bd4e9118c69d6ab531713c7212a93d50fedd1970ba439c04a00: Status 404 returned error can't find the container with id 64a79a4183033bd4e9118c69d6ab531713c7212a93d50fedd1970ba439c04a00 Oct 30 12:33:34.261663 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:34.261592 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.248816364 podStartE2EDuration="3.261573182s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.756818ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:34.261088311 +0000 UTC m=+32.387889212" watchObservedRunningTime="2026-10-30 12:33:34.261573182 +0000 UTC m=+32.388374081" Oct 30 12:33:35.209554 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:35.209511 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-874fd879-b99jw" event={"ID":"98b4567f-f0d4-47a3-a8a3-3cac381e107d","Type":"ContainerStarted","Data":"64a79a4183033bd4e9118c69d6ab531713c7212a93d50fedd1970ba439c04a00"} Oct 30 12:33:37.218865 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.218825 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-874fd879-b99jw" event={"ID":"98b4567f-f0d4-47a3-a8a3-3cac381e107d","Type":"ContainerStarted","Data":"788fc52767b399d862b92f269a9f5879f3ae2c2eee914147597ef990dae41283"} Oct 30 12:33:37.219304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.218898 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-874fd879-b99jw" event={"ID":"98b4567f-f0d4-47a3-a8a3-3cac381e107d","Type":"ContainerStarted","Data":"1d0d5b6e5b34584a78f40efc35ef4ca93c01bd07cb2b423815c1424dc36d41d5"} Oct 30 12:33:37.219304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.218928 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-874fd879-b99jw" event={"ID":"98b4567f-f0d4-47a3-a8a3-3cac381e107d","Type":"ContainerStarted","Data":"2334f9a73efda393a9150616031e75acba8858524612e653e3fd8b97450fe674"} Oct 30 12:33:37.288655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.288566 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-874fd879-b99jw" podStartSLOduration=2.223852901 podStartE2EDuration="4.288552683s" totalImagesPullingTime="2.064699782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:37.276294724 +0000 UTC m=+35.403095622" watchObservedRunningTime="2026-10-30 12:33:37.288552683 +0000 UTC m=+35.415353580" Oct 30 12:33:37.451733 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.451698 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-767dffbd8f-v7nsl"] Oct 30 12:33:37.474287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.474195 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.477024 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.476997 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-767dffbd8f-v7nsl"] Oct 30 12:33:37.530657 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.530615 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-oauth-serving-cert\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.530657 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.530666 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-config\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.530890 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.530796 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-oauth-config\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.530890 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.530851 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-trusted-ca-bundle\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.530968 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.530916 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-serving-cert\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.530968 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.530948 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-service-ca\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.531045 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.530978 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6005b7a7-0f1e-42d2-9d1e-0632531d8044-tmp\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.531081 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.531040 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnlx\" (UniqueName: \"kubernetes.io/projected/6005b7a7-0f1e-42d2-9d1e-0632531d8044-kube-api-access-fpnlx\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632082 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632048 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6005b7a7-0f1e-42d2-9d1e-0632531d8044-tmp\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632082 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632086 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnlx\" (UniqueName: \"kubernetes.io/projected/6005b7a7-0f1e-42d2-9d1e-0632531d8044-kube-api-access-fpnlx\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632402 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632113 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-oauth-serving-cert\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632402 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632145 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-config\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632402 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632322 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-oauth-config\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632402 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632383 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-trusted-ca-bundle\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632634 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632419 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-serving-cert\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632634 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632454 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-service-ca\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632634 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632502 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6005b7a7-0f1e-42d2-9d1e-0632531d8044-tmp\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.632961 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632932 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-config\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.633090 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.632973 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-oauth-serving-cert\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.633090 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.633053 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-service-ca\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.633366 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.633344 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-trusted-ca-bundle\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.634764 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.634744 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-oauth-config\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.634854 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.634829 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-serving-cert\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.655294 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.655259 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnlx\" (UniqueName: \"kubernetes.io/projected/6005b7a7-0f1e-42d2-9d1e-0632531d8044-kube-api-access-fpnlx\") pod \"console-767dffbd8f-v7nsl\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.713746 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.713706 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:37.807019 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.806905 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:37.920181 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:37.920136 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:38.125572 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:38.125537 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-767dffbd8f-v7nsl"] Oct 30 12:33:38.128108 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:33:38.128079 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6005b7a7_0f1e_42d2_9d1e_0632531d8044.slice/crio-4b4ef1c4b6aa3b554fafc60efd3f88a001b23751ebea0ed30b7c14e08f978152 WatchSource:0}: Error finding container 4b4ef1c4b6aa3b554fafc60efd3f88a001b23751ebea0ed30b7c14e08f978152: Status 404 returned error can't find the container with id 4b4ef1c4b6aa3b554fafc60efd3f88a001b23751ebea0ed30b7c14e08f978152 Oct 30 12:33:38.224172 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:38.224114 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767dffbd8f-v7nsl" event={"ID":"6005b7a7-0f1e-42d2-9d1e-0632531d8044","Type":"ContainerStarted","Data":"c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6"} Oct 30 12:33:38.224600 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:38.224188 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767dffbd8f-v7nsl" event={"ID":"6005b7a7-0f1e-42d2-9d1e-0632531d8044","Type":"ContainerStarted","Data":"4b4ef1c4b6aa3b554fafc60efd3f88a001b23751ebea0ed30b7c14e08f978152"} Oct 30 12:33:38.349024 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:38.348988 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:38.556972 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:38.556935 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:39.269831 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:39.269773 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-767dffbd8f-v7nsl" podStartSLOduration=2.269758142 podStartE2EDuration="2.269758142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:39.269380297 +0000 UTC m=+37.396181207" watchObservedRunningTime="2026-10-30 12:33:39.269758142 +0000 UTC m=+37.396559039" Oct 30 12:33:39.715065 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:39.715026 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kjjj6" Oct 30 12:33:47.807233 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:47.807193 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:47.807732 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:47.807297 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:47.813431 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:47.813403 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:48.264237 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:48.264210 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:33:48.342565 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:48.342530 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f79bf4f6d-b29lv"] Oct 30 12:33:49.263108 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.263051 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8b6c8b7bd-8d84r" podUID="8a8c4da3-8154-4b51-ac73-49afe96db85e" containerName="console" containerID="cri-o://e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a" gracePeriod=15 Oct 30 12:33:49.598000 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.597964 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:49.748377 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748344 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert\") pod \"8a8c4da3-8154-4b51-ac73-49afe96db85e\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " Oct 30 12:33:49.748377 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748392 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c\") pod \"8a8c4da3-8154-4b51-ac73-49afe96db85e\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " Oct 30 12:33:49.748593 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748430 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config\") pod \"8a8c4da3-8154-4b51-ac73-49afe96db85e\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " Oct 30 12:33:49.748593 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748469 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config\") pod \"8a8c4da3-8154-4b51-ac73-49afe96db85e\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " Oct 30 12:33:49.748593 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748513 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp\") pod \"8a8c4da3-8154-4b51-ac73-49afe96db85e\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " Oct 30 12:33:49.748593 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748567 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert\") pod \"8a8c4da3-8154-4b51-ac73-49afe96db85e\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " Oct 30 12:33:49.748797 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748597 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca\") pod \"8a8c4da3-8154-4b51-ac73-49afe96db85e\" (UID: \"8a8c4da3-8154-4b51-ac73-49afe96db85e\") " Oct 30 12:33:49.748858 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748821 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp" pod "8a8c4da3-8154-4b51-ac73-49afe96db85e" (UID: "8a8c4da3-8154-4b51-ac73-49afe96db85e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:49.748917 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748854 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config" pod "8a8c4da3-8154-4b51-ac73-49afe96db85e" (UID: "8a8c4da3-8154-4b51-ac73-49afe96db85e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:49.748995 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.748973 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert" pod "8a8c4da3-8154-4b51-ac73-49afe96db85e" (UID: "8a8c4da3-8154-4b51-ac73-49afe96db85e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:49.749193 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.749170 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca" pod "8a8c4da3-8154-4b51-ac73-49afe96db85e" (UID: "8a8c4da3-8154-4b51-ac73-49afe96db85e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:49.750795 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.750771 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert" pod "8a8c4da3-8154-4b51-ac73-49afe96db85e" (UID: "8a8c4da3-8154-4b51-ac73-49afe96db85e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:49.750892 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.750789 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config" pod "8a8c4da3-8154-4b51-ac73-49afe96db85e" (UID: "8a8c4da3-8154-4b51-ac73-49afe96db85e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:49.750892 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.750772 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c" pod "8a8c4da3-8154-4b51-ac73-49afe96db85e" (UID: "8a8c4da3-8154-4b51-ac73-49afe96db85e"). InnerVolumeSpecName "kube-api-access-vg58c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:49.850040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.849959 5515 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-serving-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.850040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.849991 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg58c\" (UniqueName: \"kubernetes.io/projected/8a8c4da3-8154-4b51-ac73-49afe96db85e-kube-api-access-vg58c\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.850040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.850003 5515 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-config\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.850040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.850012 5515 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a8c4da3-8154-4b51-ac73-49afe96db85e-console-oauth-config\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.850040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.850021 5515 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a8c4da3-8154-4b51-ac73-49afe96db85e-tmp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.850040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.850030 5515 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-oauth-serving-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.850040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:49.850040 5515 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c4da3-8154-4b51-ac73-49afe96db85e-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:33:50.269589 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:50.269560 5515 generic.go:356] "Generic (PLEG): container finished" podID="8a8c4da3-8154-4b51-ac73-49afe96db85e" containerID="e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a" exitCode=0 Oct 30 12:33:50.270002 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:50.269609 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b6c8b7bd-8d84r" Oct 30 12:33:50.270002 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:50.269620 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b6c8b7bd-8d84r" event={"ID":"8a8c4da3-8154-4b51-ac73-49afe96db85e","Type":"ContainerDied","Data":"e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a"} Oct 30 12:33:50.270002 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:50.269661 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b6c8b7bd-8d84r" event={"ID":"8a8c4da3-8154-4b51-ac73-49afe96db85e","Type":"ContainerDied","Data":"b08e46fbd7ed086401c0c4cc66fd127b646cbc6b3eb75130d8d7393753eeb95c"} Oct 30 12:33:50.270002 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:50.269679 5515 scope.go:122] "RemoveContainer" containerID="e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a" Oct 30 12:33:50.277355 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:50.277324 5515 scope.go:122] "RemoveContainer" containerID="e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a" Oct 30 12:33:50.277622 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:50.277595 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a\": container with ID starting with e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a not found: ID does not exist" containerID="e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a" Oct 30 12:33:50.277700 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:50.277629 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a"} err="failed to get container status \"e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a\": rpc error: code = NotFound desc = could not find container \"e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a\": container with ID starting with e47e0a8ee83d5cf4a0265bf99a79352fcbc161abda2c7ce1a4985f9d02709b0a not found: ID does not exist" Oct 30 12:33:50.319255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:50.319231 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b6c8b7bd-8d84r"] Oct 30 12:33:50.331280 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:50.331250 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8b6c8b7bd-8d84r"] Oct 30 12:33:51.998476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:33:51.998439 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8c4da3-8154-4b51-ac73-49afe96db85e" path="/var/lib/kubelet/pods/8a8c4da3-8154-4b51-ac73-49afe96db85e/volumes" Oct 30 12:33:58.986645 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:33:58.986592 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:34:05.335827 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:05.335788 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:05.545412 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:05.545370 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:05.750278 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:05.750236 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:13.378154 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.378108 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f79bf4f6d-b29lv" podUID="5a504de6-2016-4d68-9602-dc0ff0c2d192" containerName="console" containerID="cri-o://69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2" gracePeriod=15 Oct 30 12:34:13.629036 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.628974 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:34:13.650234 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650204 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-serving-cert\") pod \"5a504de6-2016-4d68-9602-dc0ff0c2d192\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " Oct 30 12:34:13.650430 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650251 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-config\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-config\") pod \"5a504de6-2016-4d68-9602-dc0ff0c2d192\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " Oct 30 12:34:13.650430 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650315 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-oauth-config\") pod \"5a504de6-2016-4d68-9602-dc0ff0c2d192\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " Oct 30 12:34:13.650430 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650350 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-trusted-ca-bundle\") pod \"5a504de6-2016-4d68-9602-dc0ff0c2d192\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " Oct 30 12:34:13.650430 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650388 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-service-ca\") pod \"5a504de6-2016-4d68-9602-dc0ff0c2d192\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " Oct 30 12:34:13.650430 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650426 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a504de6-2016-4d68-9602-dc0ff0c2d192-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a504de6-2016-4d68-9602-dc0ff0c2d192-tmp\") pod \"5a504de6-2016-4d68-9602-dc0ff0c2d192\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " Oct 30 12:34:13.650673 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650462 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-oauth-serving-cert\") pod \"5a504de6-2016-4d68-9602-dc0ff0c2d192\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " Oct 30 12:34:13.650673 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650518 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a504de6-2016-4d68-9602-dc0ff0c2d192-kube-api-access-x9hqn\" (UniqueName: \"kubernetes.io/projected/5a504de6-2016-4d68-9602-dc0ff0c2d192-kube-api-access-x9hqn\") pod \"5a504de6-2016-4d68-9602-dc0ff0c2d192\" (UID: \"5a504de6-2016-4d68-9602-dc0ff0c2d192\") " Oct 30 12:34:13.650771 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650736 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-config" pod "5a504de6-2016-4d68-9602-dc0ff0c2d192" (UID: "5a504de6-2016-4d68-9602-dc0ff0c2d192"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:13.650996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650842 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-service-ca" pod "5a504de6-2016-4d68-9602-dc0ff0c2d192" (UID: "5a504de6-2016-4d68-9602-dc0ff0c2d192"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:13.650996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650851 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-trusted-ca-bundle" pod "5a504de6-2016-4d68-9602-dc0ff0c2d192" (UID: "5a504de6-2016-4d68-9602-dc0ff0c2d192"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:13.650996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.650954 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a504de6-2016-4d68-9602-dc0ff0c2d192-tmp" pod "5a504de6-2016-4d68-9602-dc0ff0c2d192" (UID: "5a504de6-2016-4d68-9602-dc0ff0c2d192"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:13.651260 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.651070 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-oauth-serving-cert" pod "5a504de6-2016-4d68-9602-dc0ff0c2d192" (UID: "5a504de6-2016-4d68-9602-dc0ff0c2d192"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:13.652644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.652618 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-oauth-config" pod "5a504de6-2016-4d68-9602-dc0ff0c2d192" (UID: "5a504de6-2016-4d68-9602-dc0ff0c2d192"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:13.652743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.652647 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-serving-cert" pod "5a504de6-2016-4d68-9602-dc0ff0c2d192" (UID: "5a504de6-2016-4d68-9602-dc0ff0c2d192"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:13.652743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.652651 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a504de6-2016-4d68-9602-dc0ff0c2d192-kube-api-access-x9hqn" pod "5a504de6-2016-4d68-9602-dc0ff0c2d192" (UID: "5a504de6-2016-4d68-9602-dc0ff0c2d192"). InnerVolumeSpecName "kube-api-access-x9hqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:13.751649 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.751609 5515 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-oauth-config\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:34:13.751649 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.751641 5515 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-trusted-ca-bundle\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:34:13.751649 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.751653 5515 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:34:13.751649 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.751663 5515 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a504de6-2016-4d68-9602-dc0ff0c2d192-tmp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:34:13.751924 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.751672 5515 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-oauth-serving-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:34:13.751924 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.751681 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9hqn\" (UniqueName: \"kubernetes.io/projected/5a504de6-2016-4d68-9602-dc0ff0c2d192-kube-api-access-x9hqn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:34:13.751924 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.751689 5515 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-serving-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:34:13.751924 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:13.751697 5515 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a504de6-2016-4d68-9602-dc0ff0c2d192-console-config\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:34:14.354396 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:14.354303 5515 generic.go:356] "Generic (PLEG): container finished" podID="5a504de6-2016-4d68-9602-dc0ff0c2d192" containerID="69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2" exitCode=0 Oct 30 12:34:14.354396 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:14.354344 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f79bf4f6d-b29lv" Oct 30 12:34:14.354396 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:14.354375 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f79bf4f6d-b29lv" event={"ID":"5a504de6-2016-4d68-9602-dc0ff0c2d192","Type":"ContainerDied","Data":"69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2"} Oct 30 12:34:14.354622 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:14.354431 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f79bf4f6d-b29lv" event={"ID":"5a504de6-2016-4d68-9602-dc0ff0c2d192","Type":"ContainerDied","Data":"597a5357b2a718584e3c3955f7d6804fb8ce4119c739edd0320b849db0074e85"} Oct 30 12:34:14.354622 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:14.354452 5515 scope.go:122] "RemoveContainer" containerID="69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2" Oct 30 12:34:14.362708 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:14.362685 5515 scope.go:122] "RemoveContainer" containerID="69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2" Oct 30 12:34:14.363058 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:34:14.363035 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2\": container with ID starting with 69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2 not found: ID does not exist" containerID="69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2" Oct 30 12:34:14.363133 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:14.363068 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2"} err="failed to get container status \"69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2\": rpc error: code = NotFound desc = could not find container \"69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2\": container with ID starting with 69b74b890e0f9c5fd307c88d8e5d305ad2012540b367193106d7a85647d49ed2 not found: ID does not exist" Oct 30 12:34:14.403466 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:14.403433 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f79bf4f6d-b29lv"] Oct 30 12:34:14.415333 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:14.415296 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f79bf4f6d-b29lv"] Oct 30 12:34:15.998884 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:15.998854 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a504de6-2016-4d68-9602-dc0ff0c2d192" path="/var/lib/kubelet/pods/5a504de6-2016-4d68-9602-dc0ff0c2d192/volumes" Oct 30 12:34:27.001066 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:27.001024 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:27.232711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:27.232678 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:28.986214 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:34:28.986170 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:34:33.997570 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:33.997529 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:34.206037 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:34.206000 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:44.135304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:44.135190 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:45.327608 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.327576 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-fc75b545c-cdxqw"] Oct 30 12:34:45.327994 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.327892 5515 cpu_manager.go:414] "containerMap: removing container" podUID="8a8c4da3-8154-4b51-ac73-49afe96db85e" containerName="console" Oct 30 12:34:45.327994 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.327906 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8c4da3-8154-4b51-ac73-49afe96db85e" containerName="console" Oct 30 12:34:45.327994 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.327931 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5a504de6-2016-4d68-9602-dc0ff0c2d192" containerName="console" Oct 30 12:34:45.327994 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.327935 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a504de6-2016-4d68-9602-dc0ff0c2d192" containerName="console" Oct 30 12:34:45.327994 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.327976 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a504de6-2016-4d68-9602-dc0ff0c2d192" containerName="console" Oct 30 12:34:45.327994 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.327987 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8c4da3-8154-4b51-ac73-49afe96db85e" containerName="console" Oct 30 12:34:45.330330 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.330312 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.361132 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.361097 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc75b545c-cdxqw"] Oct 30 12:34:45.390843 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.390804 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-trusted-ca-bundle\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.390843 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.390846 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb732958-cf4e-4753-b2e8-0382a471b48e-console-oauth-config\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.391052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.390866 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb732958-cf4e-4753-b2e8-0382a471b48e-tmp\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.391052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.390926 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cmz\" (UniqueName: \"kubernetes.io/projected/eb732958-cf4e-4753-b2e8-0382a471b48e-kube-api-access-76cmz\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.391052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.391021 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb732958-cf4e-4753-b2e8-0382a471b48e-console-serving-cert\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.391153 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.391066 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-console-config\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.391153 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.391086 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-oauth-serving-cert\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.391214 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.391152 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-service-ca\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.492479 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.492440 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb732958-cf4e-4753-b2e8-0382a471b48e-console-serving-cert\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.492479 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.492481 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-console-config\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.492695 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.492509 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-oauth-serving-cert\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.492695 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.492621 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-service-ca\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.492765 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.492748 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-trusted-ca-bundle\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.492801 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.492782 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb732958-cf4e-4753-b2e8-0382a471b48e-console-oauth-config\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.492851 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.492811 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb732958-cf4e-4753-b2e8-0382a471b48e-tmp\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.493447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.492979 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cmz\" (UniqueName: \"kubernetes.io/projected/eb732958-cf4e-4753-b2e8-0382a471b48e-kube-api-access-76cmz\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.493447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.493134 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb732958-cf4e-4753-b2e8-0382a471b48e-tmp\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.493447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.493418 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-console-config\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.493711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.493477 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-oauth-serving-cert\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.493711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.493524 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-service-ca\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.493800 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.493719 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb732958-cf4e-4753-b2e8-0382a471b48e-trusted-ca-bundle\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.495243 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.495222 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb732958-cf4e-4753-b2e8-0382a471b48e-console-serving-cert\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.495319 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.495228 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb732958-cf4e-4753-b2e8-0382a471b48e-console-oauth-config\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.516248 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.516220 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cmz\" (UniqueName: \"kubernetes.io/projected/eb732958-cf4e-4753-b2e8-0382a471b48e-kube-api-access-76cmz\") pod \"console-fc75b545c-cdxqw\" (UID: \"eb732958-cf4e-4753-b2e8-0382a471b48e\") " pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.569863 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.569806 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:45.641609 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.641571 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:45.861342 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.861293 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:45.937809 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:45.937785 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc75b545c-cdxqw"] Oct 30 12:34:45.940162 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:34:45.940135 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb732958_cf4e_4753_b2e8_0382a471b48e.slice/crio-ce53fb3eed6c687f73e1e49b3b1ce1b383b369291a8991c8b7de3e24f38dc3c5 WatchSource:0}: Error finding container ce53fb3eed6c687f73e1e49b3b1ce1b383b369291a8991c8b7de3e24f38dc3c5: Status 404 returned error can't find the container with id ce53fb3eed6c687f73e1e49b3b1ce1b383b369291a8991c8b7de3e24f38dc3c5 Oct 30 12:34:46.060527 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:46.060497 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:46.465534 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:46.465488 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc75b545c-cdxqw" event={"ID":"eb732958-cf4e-4753-b2e8-0382a471b48e","Type":"ContainerStarted","Data":"50830e158341f91034c3000528d7b1ef4e4930c14f7cf0bca6bac75ebef28446"} Oct 30 12:34:46.465945 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:46.465551 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc75b545c-cdxqw" event={"ID":"eb732958-cf4e-4753-b2e8-0382a471b48e","Type":"ContainerStarted","Data":"ce53fb3eed6c687f73e1e49b3b1ce1b383b369291a8991c8b7de3e24f38dc3c5"} Oct 30 12:34:46.508147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:46.508096 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-fc75b545c-cdxqw" podStartSLOduration=1.5080812799999999 podStartE2EDuration="1.50808128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:46.507525833 +0000 UTC m=+104.634326733" watchObservedRunningTime="2026-10-30 12:34:46.50808128 +0000 UTC m=+104.634882169" Oct 30 12:34:52.216979 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:52.216941 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:52.436160 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:52.436112 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:54.994055 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:54.993986 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:55.200833 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:55.200797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:55.642084 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:55.642045 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:55.642283 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:55.642104 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:55.646792 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:55.646763 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:56.503702 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:56.503668 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fc75b545c-cdxqw" Oct 30 12:34:56.579822 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:56.579780 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-767dffbd8f-v7nsl"] Oct 30 12:34:57.361492 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:57.361454 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:57.563902 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:34:57.563869 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:58.987243 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:34:58.987204 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:35:05.696038 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:05.696000 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:07.224478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:07.224443 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:07.449746 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:07.449704 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:21.601010 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.600973 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-767dffbd8f-v7nsl" podUID="6005b7a7-0f1e-42d2-9d1e-0632531d8044" containerName="console" containerID="cri-o://c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6" gracePeriod=15 Oct 30 12:35:21.841511 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.841487 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:35:21.890524 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.890490 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-trusted-ca-bundle\") pod \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " Oct 30 12:35:21.890743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.890571 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-oauth-config\") pod \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " Oct 30 12:35:21.890743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.890599 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6005b7a7-0f1e-42d2-9d1e-0632531d8044-kube-api-access-fpnlx\" (UniqueName: \"kubernetes.io/projected/6005b7a7-0f1e-42d2-9d1e-0632531d8044-kube-api-access-fpnlx\") pod \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " Oct 30 12:35:21.890743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.890631 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-service-ca\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-service-ca\") pod \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " Oct 30 12:35:21.890743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.890665 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6005b7a7-0f1e-42d2-9d1e-0632531d8044-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6005b7a7-0f1e-42d2-9d1e-0632531d8044-tmp\") pod \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " Oct 30 12:35:21.890743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.890725 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-serving-cert\") pod \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " Oct 30 12:35:21.891108 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.890754 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-oauth-serving-cert\") pod \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " Oct 30 12:35:21.891108 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.890792 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-config\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-config\") pod \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\" (UID: \"6005b7a7-0f1e-42d2-9d1e-0632531d8044\") " Oct 30 12:35:21.891108 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.890902 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-trusted-ca-bundle" pod "6005b7a7-0f1e-42d2-9d1e-0632531d8044" (UID: "6005b7a7-0f1e-42d2-9d1e-0632531d8044"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:35:21.891108 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.891035 5515 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-trusted-ca-bundle\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:35:21.891412 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.891159 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6005b7a7-0f1e-42d2-9d1e-0632531d8044-tmp" pod "6005b7a7-0f1e-42d2-9d1e-0632531d8044" (UID: "6005b7a7-0f1e-42d2-9d1e-0632531d8044"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:35:21.891412 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.891257 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-config" pod "6005b7a7-0f1e-42d2-9d1e-0632531d8044" (UID: "6005b7a7-0f1e-42d2-9d1e-0632531d8044"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:35:21.891582 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.891555 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-service-ca" pod "6005b7a7-0f1e-42d2-9d1e-0632531d8044" (UID: "6005b7a7-0f1e-42d2-9d1e-0632531d8044"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:35:21.891662 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.891563 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-oauth-serving-cert" pod "6005b7a7-0f1e-42d2-9d1e-0632531d8044" (UID: "6005b7a7-0f1e-42d2-9d1e-0632531d8044"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:35:21.893364 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.893237 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-serving-cert" pod "6005b7a7-0f1e-42d2-9d1e-0632531d8044" (UID: "6005b7a7-0f1e-42d2-9d1e-0632531d8044"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:35:21.893364 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.893336 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-oauth-config" pod "6005b7a7-0f1e-42d2-9d1e-0632531d8044" (UID: "6005b7a7-0f1e-42d2-9d1e-0632531d8044"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:35:21.893498 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.893369 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6005b7a7-0f1e-42d2-9d1e-0632531d8044-kube-api-access-fpnlx" pod "6005b7a7-0f1e-42d2-9d1e-0632531d8044" (UID: "6005b7a7-0f1e-42d2-9d1e-0632531d8044"). InnerVolumeSpecName "kube-api-access-fpnlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:35:21.991595 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.991557 5515 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-oauth-config\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:35:21.991595 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.991586 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpnlx\" (UniqueName: \"kubernetes.io/projected/6005b7a7-0f1e-42d2-9d1e-0632531d8044-kube-api-access-fpnlx\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:35:21.991595 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.991596 5515 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:35:21.991595 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.991605 5515 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6005b7a7-0f1e-42d2-9d1e-0632531d8044-tmp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:35:21.991852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.991613 5515 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-serving-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:35:21.991852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.991622 5515 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-oauth-serving-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:35:21.991852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:21.991630 5515 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6005b7a7-0f1e-42d2-9d1e-0632531d8044-console-config\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:35:22.589372 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:22.589339 5515 generic.go:356] "Generic (PLEG): container finished" podID="6005b7a7-0f1e-42d2-9d1e-0632531d8044" containerID="c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6" exitCode=0 Oct 30 12:35:22.589561 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:22.589379 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767dffbd8f-v7nsl" event={"ID":"6005b7a7-0f1e-42d2-9d1e-0632531d8044","Type":"ContainerDied","Data":"c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6"} Oct 30 12:35:22.589561 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:22.589388 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767dffbd8f-v7nsl" Oct 30 12:35:22.589561 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:22.589417 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767dffbd8f-v7nsl" event={"ID":"6005b7a7-0f1e-42d2-9d1e-0632531d8044","Type":"ContainerDied","Data":"4b4ef1c4b6aa3b554fafc60efd3f88a001b23751ebea0ed30b7c14e08f978152"} Oct 30 12:35:22.589561 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:22.589436 5515 scope.go:122] "RemoveContainer" containerID="c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6" Oct 30 12:35:22.597241 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:22.597222 5515 scope.go:122] "RemoveContainer" containerID="c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6" Oct 30 12:35:22.597494 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:35:22.597478 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6\": container with ID starting with c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6 not found: ID does not exist" containerID="c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6" Oct 30 12:35:22.597539 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:22.597505 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6"} err="failed to get container status \"c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6\": rpc error: code = NotFound desc = could not find container \"c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6\": container with ID starting with c57046a1ecfd19766d2f8251059066ce1771b4782c015033dec6a47bbb4165f6 not found: ID does not exist" Oct 30 12:35:22.639883 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:22.639848 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-767dffbd8f-v7nsl"] Oct 30 12:35:22.651649 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:22.651622 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-767dffbd8f-v7nsl"] Oct 30 12:35:23.998965 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:23.998931 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6005b7a7-0f1e-42d2-9d1e-0632531d8044" path="/var/lib/kubelet/pods/6005b7a7-0f1e-42d2-9d1e-0632531d8044/volumes" Oct 30 12:35:26.852601 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:26.852565 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:28.729446 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:35:28.729412 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:28.985497 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:35:28.985420 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:35:58.988208 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:35:58.988165 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:36:05.873649 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:36:05.873612 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:26.234120 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:36:26.233960 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:26.446718 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:36:26.446678 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:28.864513 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:36:28.864449 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:28.986320 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:36:28.986286 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:36:58.986355 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:36:58.986316 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:37:02.505356 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:02.505311 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:02.741220 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:02.741188 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:05.928440 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:05.928398 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:10.959454 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:10.959416 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:11.180071 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.180030 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:11.858841 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.858809 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd"] Oct 30 12:37:11.859166 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.859147 5515 cpu_manager.go:414] "containerMap: removing container" podUID="6005b7a7-0f1e-42d2-9d1e-0632531d8044" containerName="console" Oct 30 12:37:11.859166 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.859164 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6005b7a7-0f1e-42d2-9d1e-0632531d8044" containerName="console" Oct 30 12:37:11.859404 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.859239 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6005b7a7-0f1e-42d2-9d1e-0632531d8044" containerName="console" Oct 30 12:37:11.861499 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.861479 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:11.890782 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.890755 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-lgfsd\"" Oct 30 12:37:11.890904 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.890825 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 12:37:11.890904 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.890826 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 12:37:11.902151 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.902130 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:11.902239 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.902169 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:11.902239 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.902193 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksrdf\" (UniqueName: \"kubernetes.io/projected/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-kube-api-access-ksrdf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:11.911586 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:11.911564 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd"] Oct 30 12:37:12.003353 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.003310 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:12.003835 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.003370 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:12.003835 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.003390 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksrdf\" (UniqueName: \"kubernetes.io/projected/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-kube-api-access-ksrdf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:12.003835 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.003673 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:12.003835 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.003743 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:12.032752 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.032720 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksrdf\" (UniqueName: \"kubernetes.io/projected/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-kube-api-access-ksrdf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:12.105576 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.105545 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2"] Oct 30 12:37:12.107859 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.107843 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.130020 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.129984 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2"] Oct 30 12:37:12.173437 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.173406 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:12.198062 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.198030 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-k7zw7\"" Oct 30 12:37:12.205255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.205220 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.205408 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.205371 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.205468 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.205421 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct2j9\" (UniqueName: \"kubernetes.io/projected/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-kube-api-access-ct2j9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.306811 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.306770 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.306993 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.306832 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.306993 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.306856 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct2j9\" (UniqueName: \"kubernetes.io/projected/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-kube-api-access-ct2j9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.307255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.307233 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.307364 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.307241 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.330633 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.330603 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct2j9\" (UniqueName: \"kubernetes.io/projected/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-kube-api-access-ct2j9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.422034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.421995 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:12.448233 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.448200 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7"] Oct 30 12:37:12.451231 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.451212 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.466526 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.466497 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 12:37:12.466663 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.466540 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 12:37:12.466796 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.466774 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 12:37:12.466918 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.466805 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 12:37:12.466918 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.466822 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 12:37:12.466996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.466938 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 12:37:12.467076 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.467062 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ljml6\"" Oct 30 12:37:12.478905 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.478881 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd"] Oct 30 12:37:12.480166 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.480097 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7"] Oct 30 12:37:12.482503 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:37:12.481798 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d506ad_5c1f_40c5_a39b_aadc66825dc5.slice/crio-0ee43120f87075bebfad8a72e5a4bb18ea2229d5ceefa63bdb19ca4ae6126ebb WatchSource:0}: Error finding container 0ee43120f87075bebfad8a72e5a4bb18ea2229d5ceefa63bdb19ca4ae6126ebb: Status 404 returned error can't find the container with id 0ee43120f87075bebfad8a72e5a4bb18ea2229d5ceefa63bdb19ca4ae6126ebb Oct 30 12:37:12.509396 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.509367 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.509561 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.509435 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.509561 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.509509 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clb6g\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-kube-api-access-clb6g\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.509561 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.509541 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.509664 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.509625 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1edc14e-1b31-46ee-889a-94e859a6bc42-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.509664 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.509659 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.509729 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.509678 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.559186 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.559149 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:12.610754 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.610718 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.610754 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.610754 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clb6g\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-kube-api-access-clb6g\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.610988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.610775 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.610988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.610800 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1edc14e-1b31-46ee-889a-94e859a6bc42-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.610988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.610822 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.610988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.610851 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.610988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.610941 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.611479 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.611456 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.613259 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.613240 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1edc14e-1b31-46ee-889a-94e859a6bc42-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.613386 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.613309 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.613598 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.613581 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.613669 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.613649 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.634469 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.634436 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.634711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.634691 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clb6g\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-kube-api-access-clb6g\") pod \"istiod-openshift-gateway-7876bfcb74-bc2h7\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.717647 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.717624 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2"] Oct 30 12:37:12.719702 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:37:12.719670 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a77c18a_29cf_463c_8a8c_d6b9a64a5355.slice/crio-e591cb36c5dd7b9ea37c20ae1f569b414473e4734603fb9f853ed262ce072e2e WatchSource:0}: Error finding container e591cb36c5dd7b9ea37c20ae1f569b414473e4734603fb9f853ed262ce072e2e: Status 404 returned error can't find the container with id e591cb36c5dd7b9ea37c20ae1f569b414473e4734603fb9f853ed262ce072e2e Oct 30 12:37:12.763448 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.763413 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:12.782829 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.782794 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:12.965627 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.965582 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" event={"ID":"3a77c18a-29cf-463c-8a8c-d6b9a64a5355","Type":"ContainerStarted","Data":"e591cb36c5dd7b9ea37c20ae1f569b414473e4734603fb9f853ed262ce072e2e"} Oct 30 12:37:12.966854 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:12.966813 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" event={"ID":"c5d506ad-5c1f-40c5-a39b-aadc66825dc5","Type":"ContainerStarted","Data":"0ee43120f87075bebfad8a72e5a4bb18ea2229d5ceefa63bdb19ca4ae6126ebb"} Oct 30 12:37:13.070453 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:13.070423 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7"] Oct 30 12:37:13.072874 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:37:13.072840 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1edc14e_1b31_46ee_889a_94e859a6bc42.slice/crio-6abf1c7e55b3de79c170f97d75ba56d97b115f4461bdb3bad32b3891258549dd WatchSource:0}: Error finding container 6abf1c7e55b3de79c170f97d75ba56d97b115f4461bdb3bad32b3891258549dd: Status 404 returned error can't find the container with id 6abf1c7e55b3de79c170f97d75ba56d97b115f4461bdb3bad32b3891258549dd Oct 30 12:37:13.866011 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:13.865930 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:13.976988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:13.976908 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" event={"ID":"f1edc14e-1b31-46ee-889a-94e859a6bc42","Type":"ContainerStarted","Data":"6abf1c7e55b3de79c170f97d75ba56d97b115f4461bdb3bad32b3891258549dd"} Oct 30 12:37:16.161146 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:16.161103 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:37:16.161631 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:16.161170 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:37:16.954186 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:16.954145 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:16.995155 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:16.995118 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" event={"ID":"f1edc14e-1b31-46ee-889a-94e859a6bc42","Type":"ContainerStarted","Data":"4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287"} Oct 30 12:37:16.995363 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:16.995199 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:17.065552 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:17.065495 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" podStartSLOduration=1.980599089 podStartE2EDuration="5.065478283s" totalImagesPullingTime="3.084879194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:17.052491002 +0000 UTC m=+255.179291899" watchObservedRunningTime="2026-10-30 12:37:17.065478283 +0000 UTC m=+255.192279182" Oct 30 12:37:17.176676 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:17.176588 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:18.002773 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:18.002742 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:37:18.305005 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:18.304725 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:18.529502 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:18.529449 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:18.740998 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:18.740957 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:19.007390 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:19.007314 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" event={"ID":"c5d506ad-5c1f-40c5-a39b-aadc66825dc5","Type":"ContainerStarted","Data":"9ebaac2b917cda5173c5cb6a01563981374873bd9f3b47d97f5b3e735c952821"} Oct 30 12:37:20.012134 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:20.012097 5515 generic.go:356] "Generic (PLEG): container finished" podID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerID="a57238f0a437dba9607f6146c54a67736e17b3917247dce8f5e0e940fc0de5ee" exitCode=0 Oct 30 12:37:20.012618 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:20.012188 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" event={"ID":"3a77c18a-29cf-463c-8a8c-d6b9a64a5355","Type":"ContainerDied","Data":"a57238f0a437dba9607f6146c54a67736e17b3917247dce8f5e0e940fc0de5ee"} Oct 30 12:37:20.013882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:20.013650 5515 generic.go:356] "Generic (PLEG): container finished" podID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerID="9ebaac2b917cda5173c5cb6a01563981374873bd9f3b47d97f5b3e735c952821" exitCode=0 Oct 30 12:37:20.013882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:20.013715 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" event={"ID":"c5d506ad-5c1f-40c5-a39b-aadc66825dc5","Type":"ContainerDied","Data":"9ebaac2b917cda5173c5cb6a01563981374873bd9f3b47d97f5b3e735c952821"} Oct 30 12:37:21.020126 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:21.019744 5515 generic.go:356] "Generic (PLEG): container finished" podID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerID="6d1b65f5699b0ac43244916b25ae4dd12b83b00c94ba6fe8529e2767574407df" exitCode=0 Oct 30 12:37:21.020126 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:21.019869 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" event={"ID":"3a77c18a-29cf-463c-8a8c-d6b9a64a5355","Type":"ContainerDied","Data":"6d1b65f5699b0ac43244916b25ae4dd12b83b00c94ba6fe8529e2767574407df"} Oct 30 12:37:21.494450 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:21.494413 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:22.026117 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:22.026081 5515 generic.go:356] "Generic (PLEG): container finished" podID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerID="c507fa81659f300da4434c371b672b7ddbef7786f6b256c197df012832f5ec1d" exitCode=0 Oct 30 12:37:22.026613 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:22.026111 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" event={"ID":"c5d506ad-5c1f-40c5-a39b-aadc66825dc5","Type":"ContainerDied","Data":"c507fa81659f300da4434c371b672b7ddbef7786f6b256c197df012832f5ec1d"} Oct 30 12:37:28.055754 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:28.055656 5515 generic.go:356] "Generic (PLEG): container finished" podID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerID="73c0a64271c4561ca569459e8d819a8c0f02df755ae9dfd59917bff1b44e0446" exitCode=0 Oct 30 12:37:28.055754 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:28.055730 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" event={"ID":"3a77c18a-29cf-463c-8a8c-d6b9a64a5355","Type":"ContainerDied","Data":"73c0a64271c4561ca569459e8d819a8c0f02df755ae9dfd59917bff1b44e0446"} Oct 30 12:37:28.987337 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:37:28.987237 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:37:29.062084 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.062045 5515 generic.go:356] "Generic (PLEG): container finished" podID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerID="0e4d4ec645a0484025498f6a40f35d3f226b4fc0828b266d18882ceacdad4406" exitCode=0 Oct 30 12:37:29.062505 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.062136 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" event={"ID":"c5d506ad-5c1f-40c5-a39b-aadc66825dc5","Type":"ContainerDied","Data":"0e4d4ec645a0484025498f6a40f35d3f226b4fc0828b266d18882ceacdad4406"} Oct 30 12:37:29.184038 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.184006 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:29.266554 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.266450 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-util\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-util\") pod \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " Oct 30 12:37:29.266554 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.266496 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-bundle\") pod \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " Oct 30 12:37:29.266803 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.266580 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-kube-api-access-ct2j9\" (UniqueName: \"kubernetes.io/projected/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-kube-api-access-ct2j9\") pod \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\" (UID: \"3a77c18a-29cf-463c-8a8c-d6b9a64a5355\") " Oct 30 12:37:29.267127 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.267099 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-bundle" pod "3a77c18a-29cf-463c-8a8c-d6b9a64a5355" (UID: "3a77c18a-29cf-463c-8a8c-d6b9a64a5355"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:29.268893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.268862 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-kube-api-access-ct2j9" pod "3a77c18a-29cf-463c-8a8c-d6b9a64a5355" (UID: "3a77c18a-29cf-463c-8a8c-d6b9a64a5355"). InnerVolumeSpecName "kube-api-access-ct2j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:29.271027 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.271001 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-util" pod "3a77c18a-29cf-463c-8a8c-d6b9a64a5355" (UID: "3a77c18a-29cf-463c-8a8c-d6b9a64a5355"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:29.367748 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.367705 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct2j9\" (UniqueName: \"kubernetes.io/projected/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-kube-api-access-ct2j9\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.367748 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.367740 5515 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-util\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:37:29.367748 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:29.367753 5515 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a77c18a-29cf-463c-8a8c-d6b9a64a5355-bundle\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:37:30.068141 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.068109 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" event={"ID":"3a77c18a-29cf-463c-8a8c-d6b9a64a5355","Type":"ContainerDied","Data":"e591cb36c5dd7b9ea37c20ae1f569b414473e4734603fb9f853ed262ce072e2e"} Oct 30 12:37:30.068498 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.068150 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dsjg2" Oct 30 12:37:30.068498 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.068158 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e591cb36c5dd7b9ea37c20ae1f569b414473e4734603fb9f853ed262ce072e2e" Oct 30 12:37:30.190277 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.190229 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:30.274485 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.274387 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-util\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-util\") pod \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " Oct 30 12:37:30.274641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.274503 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-bundle\") pod \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " Oct 30 12:37:30.274641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.274544 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-kube-api-access-ksrdf\" (UniqueName: \"kubernetes.io/projected/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-kube-api-access-ksrdf\") pod \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\" (UID: \"c5d506ad-5c1f-40c5-a39b-aadc66825dc5\") " Oct 30 12:37:30.275089 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.275061 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-bundle" pod "c5d506ad-5c1f-40c5-a39b-aadc66825dc5" (UID: "c5d506ad-5c1f-40c5-a39b-aadc66825dc5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:30.276670 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.276643 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-kube-api-access-ksrdf" pod "c5d506ad-5c1f-40c5-a39b-aadc66825dc5" (UID: "c5d506ad-5c1f-40c5-a39b-aadc66825dc5"). InnerVolumeSpecName "kube-api-access-ksrdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:30.278255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.278230 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-util" pod "c5d506ad-5c1f-40c5-a39b-aadc66825dc5" (UID: "c5d506ad-5c1f-40c5-a39b-aadc66825dc5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:37:30.375133 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.375096 5515 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-bundle\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:37:30.375133 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.375128 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksrdf\" (UniqueName: \"kubernetes.io/projected/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-kube-api-access-ksrdf\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:37:30.375133 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:30.375140 5515 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5d506ad-5c1f-40c5-a39b-aadc66825dc5-util\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:37:31.073909 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:31.073878 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" event={"ID":"c5d506ad-5c1f-40c5-a39b-aadc66825dc5","Type":"ContainerDied","Data":"0ee43120f87075bebfad8a72e5a4bb18ea2229d5ceefa63bdb19ca4ae6126ebb"} Oct 30 12:37:31.074317 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:31.073914 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2jlmd" Oct 30 12:37:31.074317 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:31.073921 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee43120f87075bebfad8a72e5a4bb18ea2229d5ceefa63bdb19ca4ae6126ebb" Oct 30 12:37:50.613507 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:50.613470 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:50.945658 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:50.945617 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:51.175160 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.175128 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:51.633637 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.633598 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:51.778457 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778421 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs"] Oct 30 12:37:51.778769 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778754 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerName="util" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778771 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerName="util" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778786 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerName="pull" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778791 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerName="pull" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778799 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerName="extract" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778804 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerName="extract" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778819 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerName="pull" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778825 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerName="pull" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778837 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerName="extract" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778841 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerName="extract" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778855 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerName="util" Oct 30 12:37:51.778855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778859 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerName="util" Oct 30 12:37:51.779207 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778906 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d506ad-5c1f-40c5-a39b-aadc66825dc5" containerName="extract" Oct 30 12:37:51.779207 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.778919 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a77c18a-29cf-463c-8a8c-d6b9a64a5355" containerName="extract" Oct 30 12:37:51.781520 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.781498 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.847340 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.847310 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 12:37:51.847340 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.847326 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 12:37:51.847340 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.847326 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 12:37:51.847577 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.847310 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 12:37:51.847577 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.847387 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n948z\"" Oct 30 12:37:51.847577 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.847455 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 12:37:51.848024 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.847993 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.848162 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.848041 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.848162 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.848154 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-manifests\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.848240 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.848178 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.848240 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.848202 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhwc\" (UniqueName: \"kubernetes.io/projected/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-kube-api-access-bjhwc\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.848366 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.848344 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-tmp\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.848518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.848501 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs"] Oct 30 12:37:51.903672 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.903566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:51.949163 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.949126 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.949163 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.949167 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.949441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.949212 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-manifests\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.949441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.949231 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.949441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.949248 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhwc\" (UniqueName: \"kubernetes.io/projected/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-kube-api-access-bjhwc\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.949441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.949297 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-tmp\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.949707 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.949669 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-manifests\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.951705 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.951676 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-tmp\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.951810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.951777 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.951930 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.951907 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.951966 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.951955 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:51.974828 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:51.974799 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhwc\" (UniqueName: \"kubernetes.io/projected/e767d215-62a4-43c4-bbf5-8c39e1efa9a7-kube-api-access-bjhwc\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-pk4zs\" (UID: \"e767d215-62a4-43c4-bbf5-8c39e1efa9a7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:52.094705 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:52.094666 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:52.127659 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:52.127624 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:52.368851 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:52.368771 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:52.553745 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:52.553704 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs"] Oct 30 12:37:52.554704 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:37:52.554679 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode767d215_62a4_43c4_bbf5_8c39e1efa9a7.slice/crio-0a68d7b45263b85c06530d362d6a77c369be4173b6fdef87645561d753c5961d WatchSource:0}: Error finding container 0a68d7b45263b85c06530d362d6a77c369be4173b6fdef87645561d753c5961d: Status 404 returned error can't find the container with id 0a68d7b45263b85c06530d362d6a77c369be4173b6fdef87645561d753c5961d Oct 30 12:37:52.609152 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:52.609116 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:53.158079 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:53.157998 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" event={"ID":"e767d215-62a4-43c4-bbf5-8c39e1efa9a7","Type":"ContainerStarted","Data":"0a68d7b45263b85c06530d362d6a77c369be4173b6fdef87645561d753c5961d"} Oct 30 12:37:54.235187 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:54.235150 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:54.457480 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:54.457441 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:54.832686 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:54.832591 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:55.054066 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:55.054022 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:56.171961 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:56.171924 5515 generic.go:356] "Generic (PLEG): container finished" podID="e767d215-62a4-43c4-bbf5-8c39e1efa9a7" containerID="aab015fe9dfc0362e5a95b9edbcd8a1da44d647c9f5975557f6a0dddd403262e" exitCode=0 Oct 30 12:37:56.172371 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:56.171974 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" event={"ID":"e767d215-62a4-43c4-bbf5-8c39e1efa9a7","Type":"ContainerDied","Data":"aab015fe9dfc0362e5a95b9edbcd8a1da44d647c9f5975557f6a0dddd403262e"} Oct 30 12:37:57.181231 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:57.181184 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" event={"ID":"e767d215-62a4-43c4-bbf5-8c39e1efa9a7","Type":"ContainerStarted","Data":"f0a3fd271df37b25e7ba3ea8bf744bc0f2e9c6003a6db30352bc060685fea897"} Oct 30 12:37:57.181650 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:57.181479 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:37:57.290384 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:37:57.290326 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" podStartSLOduration=2.926497016 podStartE2EDuration="6.290312496s" totalImagesPullingTime="2.869079704s" totalInitContainerRuntime="494.735776ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:57.278225077 +0000 UTC m=+295.405025974" watchObservedRunningTime="2026-10-30 12:37:57.290312496 +0000 UTC m=+295.417113385" Oct 30 12:37:58.986936 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:37:58.986892 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:38:00.649058 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:00.649020 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:00.880739 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:00.880700 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:02.420632 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:02.420591 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:06.121836 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:06.121800 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:06.373782 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:06.373695 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:07.794128 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:07.794086 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:08.187345 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:08.187316 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-pk4zs" Oct 30 12:38:09.610538 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:09.610497 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:09.958104 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:09.958069 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:10.243253 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:10.243171 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:12.236721 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:12.236687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:12.463416 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:12.463381 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:12.705116 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:12.705070 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:12.933408 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:12.933375 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:13.482393 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:13.482358 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:15.052574 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:15.052528 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:15.965586 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:15.965549 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:16.198228 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:16.198193 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:16.934455 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:16.934419 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:17.219469 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:17.219381 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:18.766959 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:18.766922 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:19.455440 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:19.455400 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:19.684657 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:19.684624 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:20.472794 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:20.472759 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:20.789830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:20.789745 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:21.191518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:21.191482 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:23.068801 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:23.068762 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:23.349737 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:23.349651 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:24.338475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:24.338432 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:24.633368 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:24.633333 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:24.901498 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:24.901418 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:25.100766 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.100733 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59688578c6-wtg9m"] Oct 30 12:38:25.103232 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.103211 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.128753 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.128713 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59688578c6-wtg9m"] Oct 30 12:38:25.139346 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.139323 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 12:38:25.139664 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.139642 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vnj47\"" Oct 30 12:38:25.139799 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.139756 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 12:38:25.160425 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.160354 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:25.230412 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.230375 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnp4\" (UniqueName: \"kubernetes.io/projected/b98339fa-aad1-4672-9cc8-5c09532a01e2-kube-api-access-spnp4\") pod \"kube-auth-proxy-59688578c6-wtg9m\" (UID: \"b98339fa-aad1-4672-9cc8-5c09532a01e2\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.230582 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.230427 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98339fa-aad1-4672-9cc8-5c09532a01e2-tls-certs\") pod \"kube-auth-proxy-59688578c6-wtg9m\" (UID: \"b98339fa-aad1-4672-9cc8-5c09532a01e2\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.230582 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.230560 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b98339fa-aad1-4672-9cc8-5c09532a01e2-tmp\") pod \"kube-auth-proxy-59688578c6-wtg9m\" (UID: \"b98339fa-aad1-4672-9cc8-5c09532a01e2\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.332021 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.331983 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98339fa-aad1-4672-9cc8-5c09532a01e2-tls-certs\") pod \"kube-auth-proxy-59688578c6-wtg9m\" (UID: \"b98339fa-aad1-4672-9cc8-5c09532a01e2\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.332205 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.332089 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b98339fa-aad1-4672-9cc8-5c09532a01e2-tmp\") pod \"kube-auth-proxy-59688578c6-wtg9m\" (UID: \"b98339fa-aad1-4672-9cc8-5c09532a01e2\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.332205 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.332136 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnp4\" (UniqueName: \"kubernetes.io/projected/b98339fa-aad1-4672-9cc8-5c09532a01e2-kube-api-access-spnp4\") pod \"kube-auth-proxy-59688578c6-wtg9m\" (UID: \"b98339fa-aad1-4672-9cc8-5c09532a01e2\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.334540 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.334506 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b98339fa-aad1-4672-9cc8-5c09532a01e2-tmp\") pod \"kube-auth-proxy-59688578c6-wtg9m\" (UID: \"b98339fa-aad1-4672-9cc8-5c09532a01e2\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.334667 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.334544 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98339fa-aad1-4672-9cc8-5c09532a01e2-tls-certs\") pod \"kube-auth-proxy-59688578c6-wtg9m\" (UID: \"b98339fa-aad1-4672-9cc8-5c09532a01e2\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.379025 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.378961 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnp4\" (UniqueName: \"kubernetes.io/projected/b98339fa-aad1-4672-9cc8-5c09532a01e2-kube-api-access-spnp4\") pod \"kube-auth-proxy-59688578c6-wtg9m\" (UID: \"b98339fa-aad1-4672-9cc8-5c09532a01e2\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.393764 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.393733 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:25.414052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.413977 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" Oct 30 12:38:25.596249 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.596202 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:25.711769 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.711745 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59688578c6-wtg9m"] Oct 30 12:38:25.713259 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:38:25.713231 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98339fa_aad1_4672_9cc8_5c09532a01e2.slice/crio-630414acab63c2ffc286cac4c16e10f2b349900619c5a951d5fbf84c05995c7e WatchSource:0}: Error finding container 630414acab63c2ffc286cac4c16e10f2b349900619c5a951d5fbf84c05995c7e: Status 404 returned error can't find the container with id 630414acab63c2ffc286cac4c16e10f2b349900619c5a951d5fbf84c05995c7e Oct 30 12:38:25.715175 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:25.715159 5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:38:26.301347 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:26.301302 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" event={"ID":"b98339fa-aad1-4672-9cc8-5c09532a01e2","Type":"ContainerStarted","Data":"630414acab63c2ffc286cac4c16e10f2b349900619c5a951d5fbf84c05995c7e"} Oct 30 12:38:26.891951 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:26.891911 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:27.320693 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.320623 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw"] Oct 30 12:38:27.327518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.327495 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" Oct 30 12:38:27.361282 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.361248 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bfsd7\"" Oct 30 12:38:27.401287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.401233 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw"] Oct 30 12:38:27.451170 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.451132 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 12:38:27.451170 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.451159 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 12:38:27.451444 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.451344 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 12:38:27.452749 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.452720 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6j5r\" (UniqueName: \"kubernetes.io/projected/3abefab9-6297-457f-b668-3282fd99560b-kube-api-access-s6j5r\") pod \"dns-operator-controller-manager-8557cbdb84-77jxw\" (UID: \"3abefab9-6297-457f-b668-3282fd99560b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" Oct 30 12:38:27.554054 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.554019 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6j5r\" (UniqueName: \"kubernetes.io/projected/3abefab9-6297-457f-b668-3282fd99560b-kube-api-access-s6j5r\") pod \"dns-operator-controller-manager-8557cbdb84-77jxw\" (UID: \"3abefab9-6297-457f-b668-3282fd99560b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" Oct 30 12:38:27.587033 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.586958 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6j5r\" (UniqueName: \"kubernetes.io/projected/3abefab9-6297-457f-b668-3282fd99560b-kube-api-access-s6j5r\") pod \"dns-operator-controller-manager-8557cbdb84-77jxw\" (UID: \"3abefab9-6297-457f-b668-3282fd99560b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" Oct 30 12:38:27.641301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.641249 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" Oct 30 12:38:27.691894 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.691859 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:27.967403 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:27.967366 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw"] Oct 30 12:38:27.971954 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:38:27.971920 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abefab9_6297_457f_b668_3282fd99560b.slice/crio-37b7220565494779d1a45698b4e8349947c07ac3dc7c627ebeafec1f379bba3b WatchSource:0}: Error finding container 37b7220565494779d1a45698b4e8349947c07ac3dc7c627ebeafec1f379bba3b: Status 404 returned error can't find the container with id 37b7220565494779d1a45698b4e8349947c07ac3dc7c627ebeafec1f379bba3b Oct 30 12:38:28.312163 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:28.312068 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" event={"ID":"b98339fa-aad1-4672-9cc8-5c09532a01e2","Type":"ContainerStarted","Data":"0c66133d3a60fab31b8d8793b969956a7ab6fb89c120e606eb6acbb7a66701d2"} Oct 30 12:38:28.313346 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:28.313324 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" event={"ID":"3abefab9-6297-457f-b668-3282fd99560b","Type":"ContainerStarted","Data":"37b7220565494779d1a45698b4e8349947c07ac3dc7c627ebeafec1f379bba3b"} Oct 30 12:38:28.363532 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:28.363475 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59688578c6-wtg9m" podStartSLOduration=1.050512981 podStartE2EDuration="3.363459499s" totalImagesPullingTime="2.312946518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:28.362148961 +0000 UTC m=+326.488949859" watchObservedRunningTime="2026-10-30 12:38:28.363459499 +0000 UTC m=+326.490260396" Oct 30 12:38:28.403660 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:28.403626 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:29.009083 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:38:29.009036 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:38:29.655830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.655784 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq"] Oct 30 12:38:29.663687 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.663659 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.679947 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.679925 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-qfn2f\"" Oct 30 12:38:29.691613 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.691586 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 12:38:29.691763 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.691627 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 12:38:29.717085 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.717043 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 12:38:29.718026 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.717999 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq"] Oct 30 12:38:29.773927 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.773896 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e186430-fae4-49df-a08b-29ab2912ecd2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.774118 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.773949 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e186430-fae4-49df-a08b-29ab2912ecd2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.774118 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.774081 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e186430-fae4-49df-a08b-29ab2912ecd2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.774217 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.774135 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjzk\" (UniqueName: \"kubernetes.io/projected/3e186430-fae4-49df-a08b-29ab2912ecd2-kube-api-access-9bjzk\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.807069 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.807031 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:29.875107 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.875067 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e186430-fae4-49df-a08b-29ab2912ecd2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.875359 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.875243 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e186430-fae4-49df-a08b-29ab2912ecd2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.875359 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.875342 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjzk\" (UniqueName: \"kubernetes.io/projected/3e186430-fae4-49df-a08b-29ab2912ecd2-kube-api-access-9bjzk\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.875492 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.875381 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e186430-fae4-49df-a08b-29ab2912ecd2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.875566 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.875536 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e186430-fae4-49df-a08b-29ab2912ecd2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.875727 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.875703 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e186430-fae4-49df-a08b-29ab2912ecd2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.876667 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.876646 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e186430-fae4-49df-a08b-29ab2912ecd2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.903965 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.903929 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjzk\" (UniqueName: \"kubernetes.io/projected/3e186430-fae4-49df-a08b-29ab2912ecd2-kube-api-access-9bjzk\") pod \"kserve-module-controller-manager-7566cf5fc8-gr5gq\" (UID: \"3e186430-fae4-49df-a08b-29ab2912ecd2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:29.976744 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:29.976667 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:30.009863 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:30.009819 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:30.301798 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:30.301716 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq"] Oct 30 12:38:30.305582 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:38:30.305550 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e186430_fae4_49df_a08b_29ab2912ecd2.slice/crio-46ef3d75582e97017821cea523e31ea51cfe277b82c79ca11d631fff0b7da8a6 WatchSource:0}: Error finding container 46ef3d75582e97017821cea523e31ea51cfe277b82c79ca11d631fff0b7da8a6: Status 404 returned error can't find the container with id 46ef3d75582e97017821cea523e31ea51cfe277b82c79ca11d631fff0b7da8a6 Oct 30 12:38:30.325951 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:30.325916 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" event={"ID":"3e186430-fae4-49df-a08b-29ab2912ecd2","Type":"ContainerStarted","Data":"46ef3d75582e97017821cea523e31ea51cfe277b82c79ca11d631fff0b7da8a6"} Oct 30 12:38:31.333411 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:31.333374 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" event={"ID":"3abefab9-6297-457f-b668-3282fd99560b","Type":"ContainerStarted","Data":"616e300262c3c9fcb2abafb585373f0a7d94c54135d3be8018b0169e481d62b1"} Oct 30 12:38:31.333993 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:31.333573 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" Oct 30 12:38:31.376146 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:31.376088 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" podStartSLOduration=1.964466377 podStartE2EDuration="4.376073274s" totalImagesPullingTime="2.411606897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:31.374617126 +0000 UTC m=+329.501418046" watchObservedRunningTime="2026-10-30 12:38:31.376073274 +0000 UTC m=+329.502874173" Oct 30 12:38:32.338609 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:32.338568 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" event={"ID":"3e186430-fae4-49df-a08b-29ab2912ecd2","Type":"ContainerStarted","Data":"14bdd126cba9958a88847f89b6ca389406a2132bb40f11e644b3a5759cc9ac5e"} Oct 30 12:38:32.339010 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:32.338704 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:32.393606 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:32.393550 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" podStartSLOduration=2.159759417 podStartE2EDuration="3.393536073s" totalImagesPullingTime="1.233776656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:32.393222768 +0000 UTC m=+330.520023667" watchObservedRunningTime="2026-10-30 12:38:32.393536073 +0000 UTC m=+330.520336967" Oct 30 12:38:34.432145 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:34.432110 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:34.640179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:34.640137 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:34.866988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:34.866899 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:35.205762 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:35.205720 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:35.486057 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:35.485979 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:42.341249 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:42.341219 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-77jxw" Oct 30 12:38:43.034833 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:43.034796 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:43.260665 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:43.260627 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:45.307460 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:45.307426 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:45.514703 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:45.514675 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:49.737195 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:49.737149 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:49.943633 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:49.943601 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:53.267642 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:53.267608 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:53.344234 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:53.344205 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-gr5gq" Oct 30 12:38:53.474738 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:53.474699 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:53.683482 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:53.683446 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:53.888263 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:53.888226 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:54.180340 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:54.180298 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:54.975629 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:54.975591 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:55.179167 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:55.179128 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:58.786844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:38:58.786799 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:58.987037 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:38:58.986996 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:39:02.282097 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:02.282029 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:02.479284 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:02.479230 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:04.722252 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:04.722206 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:04.929208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:04.929171 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:06.205226 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:06.205180 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:06.415697 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:06.415660 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:08.859125 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:08.859085 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:09.090705 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:09.090667 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:19.803806 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:19.803722 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:24.716762 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.716721 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr"] Oct 30 12:39:24.722552 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.722528 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.754179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.754141 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr"] Oct 30 12:39:24.831375 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.831337 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.831375 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.831375 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.831646 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.831400 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.831646 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.831476 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6646659c-fc53-463a-936b-52950d32099e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.831646 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.831599 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6646659c-fc53-463a-936b-52950d32099e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.831646 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.831637 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klqvb\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-kube-api-access-klqvb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.831806 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.831679 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.932812 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.932779 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6646659c-fc53-463a-936b-52950d32099e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.932812 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.932816 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klqvb\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-kube-api-access-klqvb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.933090 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.932847 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.933090 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.932890 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.933090 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.932906 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.933090 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.932930 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.933090 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.932944 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6646659c-fc53-463a-936b-52950d32099e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.933742 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.933720 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6646659c-fc53-463a-936b-52950d32099e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.935473 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.935447 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.935579 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.935445 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.935674 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.935657 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6646659c-fc53-463a-936b-52950d32099e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.935753 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.935732 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.960400 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.960362 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:24.960546 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:24.960471 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klqvb\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-kube-api-access-klqvb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7qpr\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:25.033537 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.033452 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:25.341388 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.341363 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr"] Oct 30 12:39:25.346490 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:39:25.346449 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6646659c_fc53_463a_936b_52950d32099e.slice/crio-a2c4194d0734b3f5926d1522e7d762ea4b26376592c3461e09f4ca2a999c3e8f WatchSource:0}: Error finding container a2c4194d0734b3f5926d1522e7d762ea4b26376592c3461e09f4ca2a999c3e8f: Status 404 returned error can't find the container with id a2c4194d0734b3f5926d1522e7d762ea4b26376592c3461e09f4ca2a999c3e8f Oct 30 12:39:25.348470 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.348436 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:39:25.348542 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.348503 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:39:25.559600 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.559564 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:25.561435 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.561394 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" event={"ID":"6646659c-fc53-463a-936b-52950d32099e","Type":"ContainerStarted","Data":"388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2"} Oct 30 12:39:25.561593 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.561448 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" event={"ID":"6646659c-fc53-463a-936b-52950d32099e","Type":"ContainerStarted","Data":"a2c4194d0734b3f5926d1522e7d762ea4b26376592c3461e09f4ca2a999c3e8f"} Oct 30 12:39:25.561593 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.561510 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:25.612362 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.612242 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" podStartSLOduration=1.6122265759999999 podStartE2EDuration="1.612226576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:25.611410157 +0000 UTC m=+383.738211054" watchObservedRunningTime="2026-10-30 12:39:25.612226576 +0000 UTC m=+383.739027473" Oct 30 12:39:25.775332 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:25.775294 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:26.026425 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:26.026394 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:26.231812 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:26.231781 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:26.458724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:26.458678 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:26.567184 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:26.567158 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 12:39:26.654705 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:26.654676 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7"] Oct 30 12:39:26.654888 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:26.654853 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" podUID="f1edc14e-1b31-46ee-889a-94e859a6bc42" containerName="discovery" containerID="cri-o://4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287" gracePeriod=30 Oct 30 12:39:26.896405 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:26.896377 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:39:27.050857 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.050771 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-cacerts\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-cacerts\") pod \"f1edc14e-1b31-46ee-889a-94e859a6bc42\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " Oct 30 12:39:27.050857 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.050813 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-dns-cert\") pod \"f1edc14e-1b31-46ee-889a-94e859a6bc42\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " Oct 30 12:39:27.051096 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.050878 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-kubeconfig\") pod \"f1edc14e-1b31-46ee-889a-94e859a6bc42\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " Oct 30 12:39:27.051096 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.050902 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1edc14e-1b31-46ee-889a-94e859a6bc42-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1edc14e-1b31-46ee-889a-94e859a6bc42-local-certs\") pod \"f1edc14e-1b31-46ee-889a-94e859a6bc42\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " Oct 30 12:39:27.051096 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.050924 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-ca-configmap\") pod \"f1edc14e-1b31-46ee-889a-94e859a6bc42\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " Oct 30 12:39:27.051096 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.050973 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-token\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-token\") pod \"f1edc14e-1b31-46ee-889a-94e859a6bc42\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " Oct 30 12:39:27.051096 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.050990 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-kube-api-access-clb6g\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-kube-api-access-clb6g\") pod \"f1edc14e-1b31-46ee-889a-94e859a6bc42\" (UID: \"f1edc14e-1b31-46ee-889a-94e859a6bc42\") " Oct 30 12:39:27.051489 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.051456 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-ca-configmap" pod "f1edc14e-1b31-46ee-889a-94e859a6bc42" (UID: "f1edc14e-1b31-46ee-889a-94e859a6bc42"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:39:27.053632 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.053603 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1edc14e-1b31-46ee-889a-94e859a6bc42-local-certs" pod "f1edc14e-1b31-46ee-889a-94e859a6bc42" (UID: "f1edc14e-1b31-46ee-889a-94e859a6bc42"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:39:27.053743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.053604 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-token" pod "f1edc14e-1b31-46ee-889a-94e859a6bc42" (UID: "f1edc14e-1b31-46ee-889a-94e859a6bc42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:39:27.053743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.053699 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-kube-api-access-clb6g" pod "f1edc14e-1b31-46ee-889a-94e859a6bc42" (UID: "f1edc14e-1b31-46ee-889a-94e859a6bc42"). InnerVolumeSpecName "kube-api-access-clb6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:39:27.053743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.053729 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-cacerts" pod "f1edc14e-1b31-46ee-889a-94e859a6bc42" (UID: "f1edc14e-1b31-46ee-889a-94e859a6bc42"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:39:27.053901 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.053792 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-kubeconfig" pod "f1edc14e-1b31-46ee-889a-94e859a6bc42" (UID: "f1edc14e-1b31-46ee-889a-94e859a6bc42"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:39:27.053901 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.053804 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-dns-cert" pod "f1edc14e-1b31-46ee-889a-94e859a6bc42" (UID: "f1edc14e-1b31-46ee-889a-94e859a6bc42"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:39:27.152458 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.152425 5515 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-kubeconfig\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:39:27.152458 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.152451 5515 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1edc14e-1b31-46ee-889a-94e859a6bc42-local-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:39:27.152458 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.152465 5515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-ca-configmap\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:39:27.152691 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.152479 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:39:27.152691 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.152488 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clb6g\" (UniqueName: \"kubernetes.io/projected/f1edc14e-1b31-46ee-889a-94e859a6bc42-kube-api-access-clb6g\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:39:27.152691 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.152496 5515 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-cacerts\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:39:27.152691 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.152505 5515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1edc14e-1b31-46ee-889a-94e859a6bc42-istio-csr-dns-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:39:27.570841 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.570808 5515 generic.go:356] "Generic (PLEG): container finished" podID="f1edc14e-1b31-46ee-889a-94e859a6bc42" containerID="4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287" exitCode=0 Oct 30 12:39:27.571025 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.570855 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" Oct 30 12:39:27.571025 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.570887 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" event={"ID":"f1edc14e-1b31-46ee-889a-94e859a6bc42","Type":"ContainerDied","Data":"4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287"} Oct 30 12:39:27.571025 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.570939 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7" event={"ID":"f1edc14e-1b31-46ee-889a-94e859a6bc42","Type":"ContainerDied","Data":"6abf1c7e55b3de79c170f97d75ba56d97b115f4461bdb3bad32b3891258549dd"} Oct 30 12:39:27.571025 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.570960 5515 scope.go:122] "RemoveContainer" containerID="4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287" Oct 30 12:39:27.579558 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.579539 5515 scope.go:122] "RemoveContainer" containerID="4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287" Oct 30 12:39:27.579815 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:39:27.579797 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287\": container with ID starting with 4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287 not found: ID does not exist" containerID="4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287" Oct 30 12:39:27.579865 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.579824 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287"} err="failed to get container status \"4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287\": rpc error: code = NotFound desc = could not find container \"4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287\": container with ID starting with 4b8751176098029f0b990d38877bd687fa4945f323dd71edb00290ec22ecc287 not found: ID does not exist" Oct 30 12:39:27.633364 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.633338 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7"] Oct 30 12:39:27.639820 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.639800 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bc2h7"] Oct 30 12:39:27.999458 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:27.999424 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1edc14e-1b31-46ee-889a-94e859a6bc42" path="/var/lib/kubelet/pods/f1edc14e-1b31-46ee-889a-94e859a6bc42/volumes" Oct 30 12:39:28.282997 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:28.282915 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:28.497929 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:28.497895 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:28.709587 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:28.709545 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:28.987019 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:39:28.986927 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:39:30.620208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:30.620169 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:30.825672 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:30.825635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:36.084373 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:36.084332 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:36.290244 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:39:36.290211 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:58.986779 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:39:58.986738 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:40:06.321870 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:40:06.321831 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:28.986568 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:40:28.986524 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:40:58.986565 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:40:58.986521 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:41:06.442019 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:41:06.441977 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:28.986027 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:41:28.985991 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:41:58.986476 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:41:58.986376 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:42:06.476884 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:42:06.476844 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:28.987125 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:42:28.987051 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:42:58.986808 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:42:58.986772 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:43:06.725002 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:43:06.724963 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:24.640497 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:43:24.640458 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:24.851425 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:43:24.851380 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:28.987632 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:43:28.987596 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:43:58.986300 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:43:58.986203 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:44:06.821891 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:44:06.821855 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:28.986976 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:44:28.986935 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:44:58.986446 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:44:58.986410 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:45:06.937047 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:45:06.937010 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:26.382336 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:45:26.382224 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:28.986131 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:45:28.986094 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:45:38.076670 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:45:38.076641 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:38.288653 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:45:38.288619 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:44.229792 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:45:44.229755 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:58.986982 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:45:58.986948 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:46:02.834538 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:46:02.834497 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-d44rw" podUID="5b35dcc8-6a2f-4cc9-a810-85ec056b5899" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:46:02.834921 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:46:02.834619 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:46:02.835402 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:46:02.835376 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-d44rw" podUID="5b35dcc8-6a2f-4cc9-a810-85ec056b5899" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:46:03.109539 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:46:03.109459 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:07.056206 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:46:07.056172 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:07.265350 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:46:07.265314 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:28.986364 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:46:28.986329 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:46:32.836023 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:46:32.835992 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d44rw" Oct 30 12:46:33.086643 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:46:33.086563 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:42.626355 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:46:42.626261 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:58.986321 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:46:58.986284 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:47:07.255159 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:47:07.255128 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:07.563607 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:47:07.563524 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:28.986864 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:47:28.986830 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:47:58.986551 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:47:58.986515 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:48:07.324623 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:48:07.324586 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:28.986748 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:48:28.986670 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:48:58.400285 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:48:58.400204 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:58.986192 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:48:58.986161 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:49:05.924755 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.924721 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-759np"] Oct 30 12:49:05.925154 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.925057 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f1edc14e-1b31-46ee-889a-94e859a6bc42" containerName="discovery" Oct 30 12:49:05.925154 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.925069 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1edc14e-1b31-46ee-889a-94e859a6bc42" containerName="discovery" Oct 30 12:49:05.925154 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.925132 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1edc14e-1b31-46ee-889a-94e859a6bc42" containerName="discovery" Oct 30 12:49:05.927984 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.927963 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:05.942771 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.942743 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 12:49:05.954187 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.954163 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cmshn\"" Oct 30 12:49:05.955750 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.955728 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-759np"] Oct 30 12:49:05.964011 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.963985 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg8q2\" (UniqueName: \"kubernetes.io/projected/247a19d9-dcfb-47e6-8da0-c1a274de2074-kube-api-access-cg8q2\") pod \"limitador-limitador-74dfcfb894-759np\" (UID: \"247a19d9-dcfb-47e6-8da0-c1a274de2074\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:05.964124 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:05.964107 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/247a19d9-dcfb-47e6-8da0-c1a274de2074-config-file\") pod \"limitador-limitador-74dfcfb894-759np\" (UID: \"247a19d9-dcfb-47e6-8da0-c1a274de2074\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:06.025616 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.025578 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:06.064610 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.064577 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg8q2\" (UniqueName: \"kubernetes.io/projected/247a19d9-dcfb-47e6-8da0-c1a274de2074-kube-api-access-cg8q2\") pod \"limitador-limitador-74dfcfb894-759np\" (UID: \"247a19d9-dcfb-47e6-8da0-c1a274de2074\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:06.064782 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.064650 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/247a19d9-dcfb-47e6-8da0-c1a274de2074-config-file\") pod \"limitador-limitador-74dfcfb894-759np\" (UID: \"247a19d9-dcfb-47e6-8da0-c1a274de2074\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:06.065204 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.065183 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/247a19d9-dcfb-47e6-8da0-c1a274de2074-config-file\") pod \"limitador-limitador-74dfcfb894-759np\" (UID: \"247a19d9-dcfb-47e6-8da0-c1a274de2074\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:06.065474 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.065455 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-759np"] Oct 30 12:49:06.087411 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.087392 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg8q2\" (UniqueName: \"kubernetes.io/projected/247a19d9-dcfb-47e6-8da0-c1a274de2074-kube-api-access-cg8q2\") pod \"limitador-limitador-74dfcfb894-759np\" (UID: \"247a19d9-dcfb-47e6-8da0-c1a274de2074\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:06.230679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.230577 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:06.243529 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.243507 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:06.534012 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.533988 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-759np"] Oct 30 12:49:06.535450 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:49:06.535418 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247a19d9_dcfb_47e6_8da0_c1a274de2074.slice/crio-4c771bcbacf83cd9a9170700c1a235463e06b7da747f66361a8815306d5a00b8 WatchSource:0}: Error finding container 4c771bcbacf83cd9a9170700c1a235463e06b7da747f66361a8815306d5a00b8: Status 404 returned error can't find the container with id 4c771bcbacf83cd9a9170700c1a235463e06b7da747f66361a8815306d5a00b8 Oct 30 12:49:06.537359 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.537342 5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:49:06.792975 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.792903 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg5vq"] Oct 30 12:49:06.797133 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.797112 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" Oct 30 12:49:06.813156 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.813135 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg5vq"] Oct 30 12:49:06.823742 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.823712 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vgvgv\"" Oct 30 12:49:06.871889 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.871864 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bshf\" (UniqueName: \"kubernetes.io/projected/c165464f-677c-4a79-a69f-ab535972cc70-kube-api-access-5bshf\") pod \"authorino-57f7f8659f-jg5vq\" (UID: \"c165464f-677c-4a79-a69f-ab535972cc70\") " pod="kuadrant-system/authorino-57f7f8659f-jg5vq" Oct 30 12:49:06.917529 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.917504 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" event={"ID":"247a19d9-dcfb-47e6-8da0-c1a274de2074","Type":"ContainerStarted","Data":"4c771bcbacf83cd9a9170700c1a235463e06b7da747f66361a8815306d5a00b8"} Oct 30 12:49:06.966396 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.966371 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:06.972431 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.972411 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bshf\" (UniqueName: \"kubernetes.io/projected/c165464f-677c-4a79-a69f-ab535972cc70-kube-api-access-5bshf\") pod \"authorino-57f7f8659f-jg5vq\" (UID: \"c165464f-677c-4a79-a69f-ab535972cc70\") " pod="kuadrant-system/authorino-57f7f8659f-jg5vq" Oct 30 12:49:06.996413 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:06.996390 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bshf\" (UniqueName: \"kubernetes.io/projected/c165464f-677c-4a79-a69f-ab535972cc70-kube-api-access-5bshf\") pod \"authorino-57f7f8659f-jg5vq\" (UID: \"c165464f-677c-4a79-a69f-ab535972cc70\") " pod="kuadrant-system/authorino-57f7f8659f-jg5vq" Oct 30 12:49:07.119507 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.119453 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" Oct 30 12:49:07.179414 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.173897 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:07.337116 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.337089 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:49:07.340599 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.340576 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:07.361028 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.361003 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:49:07.375753 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.375703 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xckbw\" (UniqueName: \"kubernetes.io/projected/10a1b17f-494a-432f-910c-7c8ac33ad3ad-kube-api-access-xckbw\") pod \"limitador-limitador-7c9b87d574-bd2wm\" (UID: \"10a1b17f-494a-432f-910c-7c8ac33ad3ad\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:07.375753 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.375740 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10a1b17f-494a-432f-910c-7c8ac33ad3ad-config-file\") pod \"limitador-limitador-7c9b87d574-bd2wm\" (UID: \"10a1b17f-494a-432f-910c-7c8ac33ad3ad\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:07.410493 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.409989 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg5vq"] Oct 30 12:49:07.412409 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:49:07.412380 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc165464f_677c_4a79_a69f_ab535972cc70.slice/crio-79802efc5b944dd818d6badf938b10c21398efc9a7151d2140fe194f4df05c43 WatchSource:0}: Error finding container 79802efc5b944dd818d6badf938b10c21398efc9a7151d2140fe194f4df05c43: Status 404 returned error can't find the container with id 79802efc5b944dd818d6badf938b10c21398efc9a7151d2140fe194f4df05c43 Oct 30 12:49:07.431546 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.431520 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:49:07.437710 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.437682 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:07.476840 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.476810 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xckbw\" (UniqueName: \"kubernetes.io/projected/10a1b17f-494a-432f-910c-7c8ac33ad3ad-kube-api-access-xckbw\") pod \"limitador-limitador-7c9b87d574-bd2wm\" (UID: \"10a1b17f-494a-432f-910c-7c8ac33ad3ad\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:07.476948 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.476858 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10a1b17f-494a-432f-910c-7c8ac33ad3ad-config-file\") pod \"limitador-limitador-7c9b87d574-bd2wm\" (UID: \"10a1b17f-494a-432f-910c-7c8ac33ad3ad\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:07.477683 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.477663 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10a1b17f-494a-432f-910c-7c8ac33ad3ad-config-file\") pod \"limitador-limitador-7c9b87d574-bd2wm\" (UID: \"10a1b17f-494a-432f-910c-7c8ac33ad3ad\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:07.500026 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.500003 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xckbw\" (UniqueName: \"kubernetes.io/projected/10a1b17f-494a-432f-910c-7c8ac33ad3ad-kube-api-access-xckbw\") pod \"limitador-limitador-7c9b87d574-bd2wm\" (UID: \"10a1b17f-494a-432f-910c-7c8ac33ad3ad\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:07.652663 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.652630 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:07.922156 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.922072 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" event={"ID":"c165464f-677c-4a79-a69f-ab535972cc70","Type":"ContainerStarted","Data":"79802efc5b944dd818d6badf938b10c21398efc9a7151d2140fe194f4df05c43"} Oct 30 12:49:07.923619 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.923589 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" event={"ID":"247a19d9-dcfb-47e6-8da0-c1a274de2074","Type":"ContainerStarted","Data":"7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6"} Oct 30 12:49:07.923859 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.923718 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:07.950315 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.950291 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:49:07.951969 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:49:07.951945 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a1b17f_494a_432f_910c_7c8ac33ad3ad.slice/crio-24a65f483605543f9d55272fa6977e878054e6c163960faaf411ebc639431f6e WatchSource:0}: Error finding container 24a65f483605543f9d55272fa6977e878054e6c163960faaf411ebc639431f6e: Status 404 returned error can't find the container with id 24a65f483605543f9d55272fa6977e878054e6c163960faaf411ebc639431f6e Oct 30 12:49:07.970195 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:07.970144 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" podStartSLOduration=1.755103998 podStartE2EDuration="2.970126024s" totalImagesPullingTime="1.215022026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:07.969133022 +0000 UTC m=+966.095933923" watchObservedRunningTime="2026-10-30 12:49:07.970126024 +0000 UTC m=+966.096926923" Oct 30 12:49:08.931347 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:08.931310 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" event={"ID":"10a1b17f-494a-432f-910c-7c8ac33ad3ad","Type":"ContainerStarted","Data":"e9d41cd6beb2139a52d2689610ed531d976bef9c8f38b611c7316cdcca575317"} Oct 30 12:49:08.931513 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:08.931371 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" event={"ID":"10a1b17f-494a-432f-910c-7c8ac33ad3ad","Type":"ContainerStarted","Data":"24a65f483605543f9d55272fa6977e878054e6c163960faaf411ebc639431f6e"} Oct 30 12:49:08.974977 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:08.974935 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" podStartSLOduration=1.97492242 podStartE2EDuration="1.97492242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:08.974212519 +0000 UTC m=+967.101013418" watchObservedRunningTime="2026-10-30 12:49:08.97492242 +0000 UTC m=+967.101723337" Oct 30 12:49:09.937070 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:09.937027 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" event={"ID":"c165464f-677c-4a79-a69f-ab535972cc70","Type":"ContainerStarted","Data":"2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27"} Oct 30 12:49:09.937281 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:09.937112 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:09.978509 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:09.978463 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" podStartSLOduration=2.388105305 podStartE2EDuration="3.978451271s" totalImagesPullingTime="1.590345966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:09.976935095 +0000 UTC m=+968.103735993" watchObservedRunningTime="2026-10-30 12:49:09.978451271 +0000 UTC m=+968.105252169" Oct 30 12:49:10.183225 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:10.183190 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg5vq"] Oct 30 12:49:11.945766 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:11.945696 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" podUID="c165464f-677c-4a79-a69f-ab535972cc70" containerName="authorino" containerID="cri-o://2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27" gracePeriod=30 Oct 30 12:49:12.186540 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.186514 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" Oct 30 12:49:12.221134 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.221072 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c165464f-677c-4a79-a69f-ab535972cc70-kube-api-access-5bshf\" (UniqueName: \"kubernetes.io/projected/c165464f-677c-4a79-a69f-ab535972cc70-kube-api-access-5bshf\") pod \"c165464f-677c-4a79-a69f-ab535972cc70\" (UID: \"c165464f-677c-4a79-a69f-ab535972cc70\") " Oct 30 12:49:12.223148 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.223123 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c165464f-677c-4a79-a69f-ab535972cc70-kube-api-access-5bshf" pod "c165464f-677c-4a79-a69f-ab535972cc70" (UID: "c165464f-677c-4a79-a69f-ab535972cc70"). InnerVolumeSpecName "kube-api-access-5bshf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:12.321642 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.321611 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bshf\" (UniqueName: \"kubernetes.io/projected/c165464f-677c-4a79-a69f-ab535972cc70-kube-api-access-5bshf\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:49:12.950710 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.950679 5515 generic.go:356] "Generic (PLEG): container finished" podID="c165464f-677c-4a79-a69f-ab535972cc70" containerID="2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27" exitCode=0 Oct 30 12:49:12.951107 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.950725 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" Oct 30 12:49:12.951107 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.950765 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" event={"ID":"c165464f-677c-4a79-a69f-ab535972cc70","Type":"ContainerDied","Data":"2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27"} Oct 30 12:49:12.951107 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.950809 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jg5vq" event={"ID":"c165464f-677c-4a79-a69f-ab535972cc70","Type":"ContainerDied","Data":"79802efc5b944dd818d6badf938b10c21398efc9a7151d2140fe194f4df05c43"} Oct 30 12:49:12.951107 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.950831 5515 scope.go:122] "RemoveContainer" containerID="2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27" Oct 30 12:49:12.959187 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.959166 5515 scope.go:122] "RemoveContainer" containerID="2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27" Oct 30 12:49:12.959448 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:49:12.959431 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27\": container with ID starting with 2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27 not found: ID does not exist" containerID="2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27" Oct 30 12:49:12.959497 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:12.959457 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27"} err="failed to get container status \"2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27\": rpc error: code = NotFound desc = could not find container \"2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27\": container with ID starting with 2adf0e6631af3b743386396944e55b1f16a9415db759c5125428a3af5618dc27 not found: ID does not exist" Oct 30 12:49:13.001359 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:13.001333 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg5vq"] Oct 30 12:49:13.005908 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:13.005889 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg5vq"] Oct 30 12:49:13.998445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:13.998416 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c165464f-677c-4a79-a69f-ab535972cc70" path="/var/lib/kubelet/pods/c165464f-677c-4a79-a69f-ab535972cc70/volumes" Oct 30 12:49:18.933083 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:18.933056 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:20.942939 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:20.942909 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bd2wm" Oct 30 12:49:21.010932 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:21.010905 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-759np"] Oct 30 12:49:21.011098 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:21.011073 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" podUID="247a19d9-dcfb-47e6-8da0-c1a274de2074" containerName="limitador" containerID="cri-o://7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6" gracePeriod=30 Oct 30 12:49:21.949183 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:21.949160 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:21.991442 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:21.991416 5515 generic.go:356] "Generic (PLEG): container finished" podID="247a19d9-dcfb-47e6-8da0-c1a274de2074" containerID="7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6" exitCode=0 Oct 30 12:49:21.991573 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:21.991457 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" Oct 30 12:49:21.991573 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:21.991476 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" event={"ID":"247a19d9-dcfb-47e6-8da0-c1a274de2074","Type":"ContainerDied","Data":"7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6"} Oct 30 12:49:21.991573 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:21.991543 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-759np" event={"ID":"247a19d9-dcfb-47e6-8da0-c1a274de2074","Type":"ContainerDied","Data":"4c771bcbacf83cd9a9170700c1a235463e06b7da747f66361a8815306d5a00b8"} Oct 30 12:49:21.991573 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:21.991566 5515 scope.go:122] "RemoveContainer" containerID="7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6" Oct 30 12:49:22.001426 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.001406 5515 scope.go:122] "RemoveContainer" containerID="7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6" Oct 30 12:49:22.001690 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:49:22.001672 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6\": container with ID starting with 7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6 not found: ID does not exist" containerID="7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6" Oct 30 12:49:22.001746 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.001708 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6"} err="failed to get container status \"7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6\": rpc error: code = NotFound desc = could not find container \"7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6\": container with ID starting with 7f042a89ce2802ba2ebd292cd1e33b448ddcd5ab4f18fd0cff2a6eebda1768c6 not found: ID does not exist" Oct 30 12:49:22.096372 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.096316 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247a19d9-dcfb-47e6-8da0-c1a274de2074-config-file\" (UniqueName: \"kubernetes.io/configmap/247a19d9-dcfb-47e6-8da0-c1a274de2074-config-file\") pod \"247a19d9-dcfb-47e6-8da0-c1a274de2074\" (UID: \"247a19d9-dcfb-47e6-8da0-c1a274de2074\") " Oct 30 12:49:22.096372 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.096357 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247a19d9-dcfb-47e6-8da0-c1a274de2074-kube-api-access-cg8q2\" (UniqueName: \"kubernetes.io/projected/247a19d9-dcfb-47e6-8da0-c1a274de2074-kube-api-access-cg8q2\") pod \"247a19d9-dcfb-47e6-8da0-c1a274de2074\" (UID: \"247a19d9-dcfb-47e6-8da0-c1a274de2074\") " Oct 30 12:49:22.096709 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.096684 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247a19d9-dcfb-47e6-8da0-c1a274de2074-config-file" pod "247a19d9-dcfb-47e6-8da0-c1a274de2074" (UID: "247a19d9-dcfb-47e6-8da0-c1a274de2074"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:49:22.098326 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.098305 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247a19d9-dcfb-47e6-8da0-c1a274de2074-kube-api-access-cg8q2" pod "247a19d9-dcfb-47e6-8da0-c1a274de2074" (UID: "247a19d9-dcfb-47e6-8da0-c1a274de2074"). InnerVolumeSpecName "kube-api-access-cg8q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:22.196826 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.196794 5515 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/247a19d9-dcfb-47e6-8da0-c1a274de2074-config-file\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:49:22.196826 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.196820 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg8q2\" (UniqueName: \"kubernetes.io/projected/247a19d9-dcfb-47e6-8da0-c1a274de2074-kube-api-access-cg8q2\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:49:22.337818 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.337788 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-759np"] Oct 30 12:49:22.351800 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:22.351735 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-759np"] Oct 30 12:49:23.999259 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:23.999231 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247a19d9-dcfb-47e6-8da0-c1a274de2074" path="/var/lib/kubelet/pods/247a19d9-dcfb-47e6-8da0-c1a274de2074/volumes" Oct 30 12:49:26.113790 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.113757 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:26.329602 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.329567 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:26.721229 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.721201 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt"] Oct 30 12:49:26.721556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.721543 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c165464f-677c-4a79-a69f-ab535972cc70" containerName="authorino" Oct 30 12:49:26.721603 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.721557 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c165464f-677c-4a79-a69f-ab535972cc70" containerName="authorino" Oct 30 12:49:26.721603 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.721573 5515 cpu_manager.go:414] "containerMap: removing container" podUID="247a19d9-dcfb-47e6-8da0-c1a274de2074" containerName="limitador" Oct 30 12:49:26.721603 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.721579 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247a19d9-dcfb-47e6-8da0-c1a274de2074" containerName="limitador" Oct 30 12:49:26.721692 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.721637 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c165464f-677c-4a79-a69f-ab535972cc70" containerName="authorino" Oct 30 12:49:26.721692 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.721647 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="247a19d9-dcfb-47e6-8da0-c1a274de2074" containerName="limitador" Oct 30 12:49:26.725759 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.725743 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:26.742307 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.742264 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:49:26.744942 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.744919 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:49:26.746195 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.746164 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt"] Oct 30 12:49:26.754475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.754454 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 12:49:26.754594 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.754577 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ng4f9\"" Oct 30 12:49:26.833762 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.833738 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttchg\" (UniqueName: \"kubernetes.io/projected/792621b1-c1c3-4aa5-b5cf-0b07370bd2e4-kube-api-access-ttchg\") pod \"postgres-7cdbf6dc44-f69mt\" (UID: \"792621b1-c1c3-4aa5-b5cf-0b07370bd2e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:26.833882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.833794 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/792621b1-c1c3-4aa5-b5cf-0b07370bd2e4-data\") pod \"postgres-7cdbf6dc44-f69mt\" (UID: \"792621b1-c1c3-4aa5-b5cf-0b07370bd2e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:26.934299 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.934257 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/792621b1-c1c3-4aa5-b5cf-0b07370bd2e4-data\") pod \"postgres-7cdbf6dc44-f69mt\" (UID: \"792621b1-c1c3-4aa5-b5cf-0b07370bd2e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:26.934414 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.934318 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttchg\" (UniqueName: \"kubernetes.io/projected/792621b1-c1c3-4aa5-b5cf-0b07370bd2e4-kube-api-access-ttchg\") pod \"postgres-7cdbf6dc44-f69mt\" (UID: \"792621b1-c1c3-4aa5-b5cf-0b07370bd2e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:26.934611 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.934593 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/792621b1-c1c3-4aa5-b5cf-0b07370bd2e4-data\") pod \"postgres-7cdbf6dc44-f69mt\" (UID: \"792621b1-c1c3-4aa5-b5cf-0b07370bd2e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:26.968458 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.968436 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttchg\" (UniqueName: \"kubernetes.io/projected/792621b1-c1c3-4aa5-b5cf-0b07370bd2e4-kube-api-access-ttchg\") pod \"postgres-7cdbf6dc44-f69mt\" (UID: \"792621b1-c1c3-4aa5-b5cf-0b07370bd2e4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:26.970129 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:26.970106 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:27.037888 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:27.037838 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:27.326548 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:27.326525 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt"] Oct 30 12:49:27.328451 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:49:27.328421 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792621b1_c1c3_4aa5_b5cf_0b07370bd2e4.slice/crio-9915afe8e0d6a0b7aa651fee5c714c0f2b9889ae7658ab833261ad51d227bce0 WatchSource:0}: Error finding container 9915afe8e0d6a0b7aa651fee5c714c0f2b9889ae7658ab833261ad51d227bce0: Status 404 returned error can't find the container with id 9915afe8e0d6a0b7aa651fee5c714c0f2b9889ae7658ab833261ad51d227bce0 Oct 30 12:49:28.019236 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:28.019194 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" event={"ID":"792621b1-c1c3-4aa5-b5cf-0b07370bd2e4","Type":"ContainerStarted","Data":"9915afe8e0d6a0b7aa651fee5c714c0f2b9889ae7658ab833261ad51d227bce0"} Oct 30 12:49:28.991329 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:49:28.991289 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:49:32.038458 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:32.038424 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" event={"ID":"792621b1-c1c3-4aa5-b5cf-0b07370bd2e4","Type":"ContainerStarted","Data":"4f83afa6daf785b5e44edb086723acdcb672d13e01bca1131bfbb7865ab25740"} Oct 30 12:49:32.038818 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:32.038551 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:32.079102 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:32.079058 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" podStartSLOduration=1.511015993 podStartE2EDuration="6.079046953s" totalImagesPullingTime="4.56803096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:32.078905302 +0000 UTC m=+990.205706201" watchObservedRunningTime="2026-10-30 12:49:32.079046953 +0000 UTC m=+990.205847851" Oct 30 12:49:38.071745 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:38.071712 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f69mt" Oct 30 12:49:45.364657 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:45.364586 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:45.569811 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:45.569774 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:45.782977 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:45.782941 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:48.794637 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:48.794603 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:49.009405 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:49.009373 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:50.907755 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:50.907724 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:51.121745 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:51.121715 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:55.711036 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:49:55.711003 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:58.986301 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:49:58.986254 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:50:07.575770 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:07.575734 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:18.561115 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:18.561077 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:18.770385 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:18.770345 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:19.998620 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:19.998590 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:20.149205 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.149175 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj"] Oct 30 12:50:20.151559 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.151541 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.182373 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.182343 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj"] Oct 30 12:50:20.203379 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.203348 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:20.228478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.228453 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dlqhq\"" Oct 30 12:50:20.228637 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.228496 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 12:50:20.228637 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.228554 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 12:50:20.256409 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.256341 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0791810-2e8a-44ca-8782-f9effdd6ffb1-openshift-service-ca\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.256409 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.256371 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-maas-api-tls\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.256409 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.256393 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8bmb\" (UniqueName: \"kubernetes.io/projected/d0791810-2e8a-44ca-8782-f9effdd6ffb1-kube-api-access-h8bmb\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.256607 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.256504 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-metrics-certs\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.357644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.357614 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-metrics-certs\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.357774 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.357667 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0791810-2e8a-44ca-8782-f9effdd6ffb1-openshift-service-ca\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.357774 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.357695 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-maas-api-tls\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.357774 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.357716 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8bmb\" (UniqueName: \"kubernetes.io/projected/d0791810-2e8a-44ca-8782-f9effdd6ffb1-kube-api-access-h8bmb\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.358437 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.358414 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0791810-2e8a-44ca-8782-f9effdd6ffb1-openshift-service-ca\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.359956 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.359935 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-maas-api-tls\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.360131 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.360110 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-metrics-certs\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.397528 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.397498 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8bmb\" (UniqueName: \"kubernetes.io/projected/d0791810-2e8a-44ca-8782-f9effdd6ffb1-kube-api-access-h8bmb\") pod \"maas-api-fc7847f9-gcvqj\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.463531 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.463503 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:20.755750 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:20.755723 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj"] Oct 30 12:50:20.758388 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:50:20.758361 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0791810_2e8a_44ca_8782_f9effdd6ffb1.slice/crio-956e7c4665c0fa631e3b394fbe25070d03c421227a7b06db7e46335442577c91 WatchSource:0}: Error finding container 956e7c4665c0fa631e3b394fbe25070d03c421227a7b06db7e46335442577c91: Status 404 returned error can't find the container with id 956e7c4665c0fa631e3b394fbe25070d03c421227a7b06db7e46335442577c91 Oct 30 12:50:21.246098 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:21.246065 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" event={"ID":"d0791810-2e8a-44ca-8782-f9effdd6ffb1","Type":"ContainerStarted","Data":"956e7c4665c0fa631e3b394fbe25070d03c421227a7b06db7e46335442577c91"} Oct 30 12:50:22.207826 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:22.207793 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:50:23.256355 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:23.256324 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" event={"ID":"d0791810-2e8a-44ca-8782-f9effdd6ffb1","Type":"ContainerStarted","Data":"a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097"} Oct 30 12:50:23.256735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:23.256432 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:23.302802 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:23.302719 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" podStartSLOduration=1.030072885 podStartE2EDuration="3.302706675s" totalImagesPullingTime="2.27263379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:23.301196677 +0000 UTC m=+1041.427997575" watchObservedRunningTime="2026-10-30 12:50:23.302706675 +0000 UTC m=+1041.429507572" Oct 30 12:50:28.986461 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:50:28.986425 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:50:29.266032 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:29.265967 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:29.827743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:29.827711 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:30.047160 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.047125 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:30.274864 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.274835 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-l8w2f"] Oct 30 12:50:30.277675 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.277655 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:30.304601 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.304578 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vgvgv\"" Oct 30 12:50:30.304890 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.304874 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 12:50:30.306622 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.306602 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-l8w2f"] Oct 30 12:50:30.344350 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.344327 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-tls-cert\") pod \"authorino-6889cbd8c4-l8w2f\" (UID: \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\") " pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:30.344476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.344356 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bfp8\" (UniqueName: \"kubernetes.io/projected/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-kube-api-access-4bfp8\") pod \"authorino-6889cbd8c4-l8w2f\" (UID: \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\") " pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:30.399222 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.399197 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-l8w2f"] Oct 30 12:50:30.399400 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:50:30.399383 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4bfp8 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" podUID="d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d" Oct 30 12:50:30.444835 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.444810 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-tls-cert\") pod \"authorino-6889cbd8c4-l8w2f\" (UID: \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\") " pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:30.444941 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.444842 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bfp8\" (UniqueName: \"kubernetes.io/projected/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-kube-api-access-4bfp8\") pod \"authorino-6889cbd8c4-l8w2f\" (UID: \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\") " pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:30.447164 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.447139 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-tls-cert\") pod \"authorino-6889cbd8c4-l8w2f\" (UID: \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\") " pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:30.468120 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:30.468101 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bfp8\" (UniqueName: \"kubernetes.io/projected/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-kube-api-access-4bfp8\") pod \"authorino-6889cbd8c4-l8w2f\" (UID: \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\") " pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:31.289557 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:31.289523 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:31.295622 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:31.295598 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:31.455259 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:31.455221 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-tls-cert\" (UniqueName: \"kubernetes.io/secret/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-tls-cert\") pod \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\" (UID: \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\") " Oct 30 12:50:31.455259 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:31.455286 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-kube-api-access-4bfp8\" (UniqueName: \"kubernetes.io/projected/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-kube-api-access-4bfp8\") pod \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\" (UID: \"d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d\") " Oct 30 12:50:31.457233 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:31.457206 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-tls-cert" pod "d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d" (UID: "d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:50:31.457345 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:31.457250 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-kube-api-access-4bfp8" pod "d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d" (UID: "d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d"). InnerVolumeSpecName "kube-api-access-4bfp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:31.556227 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:31.556152 5515 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-tls-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:50:31.556227 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:31.556188 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bfp8\" (UniqueName: \"kubernetes.io/projected/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d-kube-api-access-4bfp8\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:50:32.293757 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:32.293667 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-l8w2f" Oct 30 12:50:32.373889 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:32.373856 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-l8w2f"] Oct 30 12:50:32.378866 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:32.378839 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-l8w2f"] Oct 30 12:50:33.998713 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:33.998683 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d" path="/var/lib/kubelet/pods/d7fd8cf5-d57d-44f7-8eb4-0d5ea837c12d/volumes" Oct 30 12:50:35.522356 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.522323 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v"] Oct 30 12:50:35.524829 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.524811 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.553801 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.553762 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v"] Oct 30 12:50:35.624700 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.624673 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tcq9j\"" Oct 30 12:50:35.690247 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.690218 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.690247 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.690249 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4053f95a-9bf2-4633-9016-219e780f7578-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.690487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.690290 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.690487 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.690435 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.690609 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.690485 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.690609 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.690534 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97wh\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-kube-api-access-z97wh\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.690609 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.690567 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.690609 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.690598 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4053f95a-9bf2-4633-9016-219e780f7578-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.690743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.690665 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.791949 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.791875 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.791949 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.791932 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.791955 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4053f95a-9bf2-4633-9016-219e780f7578-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.791990 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792032 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792064 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792102 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97wh\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-kube-api-access-z97wh\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792127 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792152 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4053f95a-9bf2-4633-9016-219e780f7578-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792571 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792342 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792651 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792627 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792720 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792800 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792741 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.792985 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.792966 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4053f95a-9bf2-4633-9016-219e780f7578-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.794234 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.794214 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.794465 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.794449 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4053f95a-9bf2-4633-9016-219e780f7578-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.815676 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.815651 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.815864 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.815849 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97wh\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-kube-api-access-z97wh\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7cq2v\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:35.835753 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:35.835735 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:36.145153 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:36.145124 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v"] Oct 30 12:50:36.147068 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:50:36.147042 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4053f95a_9bf2_4633_9016_219e780f7578.slice/crio-27415c5ee90e558adcaa41b56cee56737ab9a02968bad07a21aadea794aed370 WatchSource:0}: Error finding container 27415c5ee90e558adcaa41b56cee56737ab9a02968bad07a21aadea794aed370: Status 404 returned error can't find the container with id 27415c5ee90e558adcaa41b56cee56737ab9a02968bad07a21aadea794aed370 Oct 30 12:50:36.311058 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:36.311016 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" event={"ID":"4053f95a-9bf2-4633-9016-219e780f7578","Type":"ContainerStarted","Data":"27415c5ee90e558adcaa41b56cee56737ab9a02968bad07a21aadea794aed370"} Oct 30 12:50:38.515777 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:38.515744 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:50:38.516059 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:38.515811 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:50:38.516059 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:38.515841 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:50:39.326730 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.326692 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" event={"ID":"4053f95a-9bf2-4633-9016-219e780f7578","Type":"ContainerStarted","Data":"56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15"} Oct 30 12:50:39.384601 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.384555 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podStartSLOduration=2.018812924 podStartE2EDuration="4.384541639s" totalImagesPullingTime="2.365728715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:39.383727144 +0000 UTC m=+1057.510528041" watchObservedRunningTime="2026-10-30 12:50:39.384541639 +0000 UTC m=+1057.511342536" Oct 30 12:50:39.488175 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.488144 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj"] Oct 30 12:50:39.488441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.488394 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" podUID="d0791810-2e8a-44ca-8782-f9effdd6ffb1" containerName="maas-api" containerID="cri-o://a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097" gracePeriod=30 Oct 30 12:50:39.767495 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.767469 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:39.836530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.836489 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:39.837600 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.837571 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7cq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 12:50:39.837715 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.837624 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 12:50:39.934317 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.934288 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-maas-api-tls\") pod \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " Oct 30 12:50:39.934519 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.934368 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0791810-2e8a-44ca-8782-f9effdd6ffb1-kube-api-access-h8bmb\" (UniqueName: \"kubernetes.io/projected/d0791810-2e8a-44ca-8782-f9effdd6ffb1-kube-api-access-h8bmb\") pod \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " Oct 30 12:50:39.934519 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.934404 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-metrics-certs\") pod \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " Oct 30 12:50:39.934519 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.934431 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0791810-2e8a-44ca-8782-f9effdd6ffb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0791810-2e8a-44ca-8782-f9effdd6ffb1-openshift-service-ca\") pod \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\" (UID: \"d0791810-2e8a-44ca-8782-f9effdd6ffb1\") " Oct 30 12:50:39.934878 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.934852 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0791810-2e8a-44ca-8782-f9effdd6ffb1-openshift-service-ca" pod "d0791810-2e8a-44ca-8782-f9effdd6ffb1" (UID: "d0791810-2e8a-44ca-8782-f9effdd6ffb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:39.936478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.936448 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-maas-api-tls" pod "d0791810-2e8a-44ca-8782-f9effdd6ffb1" (UID: "d0791810-2e8a-44ca-8782-f9effdd6ffb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:50:39.936566 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.936508 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0791810-2e8a-44ca-8782-f9effdd6ffb1-kube-api-access-h8bmb" pod "d0791810-2e8a-44ca-8782-f9effdd6ffb1" (UID: "d0791810-2e8a-44ca-8782-f9effdd6ffb1"). InnerVolumeSpecName "kube-api-access-h8bmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:39.936566 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:39.936512 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-metrics-certs" pod "d0791810-2e8a-44ca-8782-f9effdd6ffb1" (UID: "d0791810-2e8a-44ca-8782-f9effdd6ffb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:50:40.035604 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.035582 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:50:40.035604 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.035604 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0791810-2e8a-44ca-8782-f9effdd6ffb1-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:50:40.035749 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.035614 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0791810-2e8a-44ca-8782-f9effdd6ffb1-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:50:40.035749 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.035622 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8bmb\" (UniqueName: \"kubernetes.io/projected/d0791810-2e8a-44ca-8782-f9effdd6ffb1-kube-api-access-h8bmb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:50:40.332512 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.332433 5515 generic.go:356] "Generic (PLEG): container finished" podID="d0791810-2e8a-44ca-8782-f9effdd6ffb1" containerID="a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097" exitCode=0 Oct 30 12:50:40.332512 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.332501 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" Oct 30 12:50:40.332687 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.332542 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" event={"ID":"d0791810-2e8a-44ca-8782-f9effdd6ffb1","Type":"ContainerDied","Data":"a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097"} Oct 30 12:50:40.332687 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.332615 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj" event={"ID":"d0791810-2e8a-44ca-8782-f9effdd6ffb1","Type":"ContainerDied","Data":"956e7c4665c0fa631e3b394fbe25070d03c421227a7b06db7e46335442577c91"} Oct 30 12:50:40.332687 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.332646 5515 scope.go:122] "RemoveContainer" containerID="a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097" Oct 30 12:50:40.341437 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.341417 5515 scope.go:122] "RemoveContainer" containerID="a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097" Oct 30 12:50:40.341672 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:50:40.341650 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097\": container with ID starting with a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097 not found: ID does not exist" containerID="a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097" Oct 30 12:50:40.341735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.341680 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097"} err="failed to get container status \"a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097\": rpc error: code = NotFound desc = could not find container \"a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097\": container with ID starting with a03c70d25eb1f6e869e47ee89679b6287d22be7e0367a5704a71770b6cb6b097 not found: ID does not exist" Oct 30 12:50:40.382594 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.382567 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj"] Oct 30 12:50:40.415289 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.415136 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-fc7847f9-gcvqj"] Oct 30 12:50:40.836647 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.836614 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7cq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 12:50:40.837010 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:40.836685 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 12:50:41.836800 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:41.836764 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7cq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 12:50:41.837179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:41.836817 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 12:50:42.021331 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:42.021301 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0791810-2e8a-44ca-8782-f9effdd6ffb1" path="/var/lib/kubelet/pods/d0791810-2e8a-44ca-8782-f9effdd6ffb1/volumes" Oct 30 12:50:42.837164 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:42.837128 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7cq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 12:50:42.837580 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:42.837177 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 12:50:43.836601 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:43.836562 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7cq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 12:50:43.836790 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:43.836625 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 12:50:44.840822 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:44.840791 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:44.841205 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:44.841077 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:44.841690 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:50:44.841672 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 12:50:58.986313 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:50:58.986263 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:51:07.696834 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.696795 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:07.772223 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.772188 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ccqqn"] Oct 30 12:51:07.772710 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.772692 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d0791810-2e8a-44ca-8782-f9effdd6ffb1" containerName="maas-api" Oct 30 12:51:07.772757 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.772712 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0791810-2e8a-44ca-8782-f9effdd6ffb1" containerName="maas-api" Oct 30 12:51:07.772791 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.772779 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0791810-2e8a-44ca-8782-f9effdd6ffb1" containerName="maas-api" Oct 30 12:51:07.779282 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.779256 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:07.795316 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.795296 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-62t9m\"" Oct 30 12:51:07.807310 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.807285 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 12:51:07.807448 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.807424 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 12:51:07.808604 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.808582 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ccqqn"] Oct 30 12:51:07.856473 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.856448 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-ccqqn\" (UID: \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:07.856628 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.856493 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxlx\" (UniqueName: \"kubernetes.io/projected/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-kube-api-access-pnxlx\") pod \"ai-gateway-controller-7d85675f98-ccqqn\" (UID: \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:07.914924 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.914894 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:07.956982 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.956919 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-ccqqn\" (UID: \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:07.956982 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.956960 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxlx\" (UniqueName: \"kubernetes.io/projected/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-kube-api-access-pnxlx\") pod \"ai-gateway-controller-7d85675f98-ccqqn\" (UID: \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:07.959352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.959322 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-ccqqn\" (UID: \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:07.980038 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:07.980016 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxlx\" (UniqueName: \"kubernetes.io/projected/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-kube-api-access-pnxlx\") pod \"ai-gateway-controller-7d85675f98-ccqqn\" (UID: \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:08.092209 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:08.092181 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:08.117685 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:08.117661 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:08.383714 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:08.383691 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ccqqn"] Oct 30 12:51:08.385443 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:51:08.385412 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecba75f_13f3_46d6_8d7a_8493e0a37c54.slice/crio-dceca723c981a62c396f9ebb0ecc17d3821caa0910f9204103e98ccbfa3f3b22 WatchSource:0}: Error finding container dceca723c981a62c396f9ebb0ecc17d3821caa0910f9204103e98ccbfa3f3b22: Status 404 returned error can't find the container with id dceca723c981a62c396f9ebb0ecc17d3821caa0910f9204103e98ccbfa3f3b22 Oct 30 12:51:08.466245 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:08.466214 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" event={"ID":"0ecba75f-13f3-46d6-8d7a-8493e0a37c54","Type":"ContainerStarted","Data":"dceca723c981a62c396f9ebb0ecc17d3821caa0910f9204103e98ccbfa3f3b22"} Oct 30 12:51:10.476718 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:10.476679 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" event={"ID":"0ecba75f-13f3-46d6-8d7a-8493e0a37c54","Type":"ContainerStarted","Data":"28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26"} Oct 30 12:51:10.477496 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:10.477473 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:10.517077 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:10.517026 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" podStartSLOduration=1.973206758 podStartE2EDuration="3.517014443s" totalImagesPullingTime="1.543807685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:10.516833629 +0000 UTC m=+1088.643634528" watchObservedRunningTime="2026-10-30 12:51:10.517014443 +0000 UTC m=+1088.643815367" Oct 30 12:51:12.165099 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.165021 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:12.256579 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.256546 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-ld25h"] Oct 30 12:51:12.259958 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.259937 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:12.274369 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.274347 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 12:51:12.285985 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.285963 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-x8zvw\"" Oct 30 12:51:12.287159 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.287136 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-ld25h"] Oct 30 12:51:12.296451 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.296428 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4951dbc-6cb4-40ce-af70-a8d331e7c2b8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-ld25h\" (UID: \"b4951dbc-6cb4-40ce-af70-a8d331e7c2b8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:12.296560 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.296484 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrtt\" (UniqueName: \"kubernetes.io/projected/b4951dbc-6cb4-40ce-af70-a8d331e7c2b8-kube-api-access-5wrtt\") pod \"payload-pre-processing-7b98bb678c-ld25h\" (UID: \"b4951dbc-6cb4-40ce-af70-a8d331e7c2b8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:12.345842 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.345809 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-bvptt"] Oct 30 12:51:12.349375 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.349352 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:12.370882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.370851 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-bvptt"] Oct 30 12:51:12.373580 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.373554 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:12.397173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.397141 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrtt\" (UniqueName: \"kubernetes.io/projected/b4951dbc-6cb4-40ce-af70-a8d331e7c2b8-kube-api-access-5wrtt\") pod \"payload-pre-processing-7b98bb678c-ld25h\" (UID: \"b4951dbc-6cb4-40ce-af70-a8d331e7c2b8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:12.397387 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.397227 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4951dbc-6cb4-40ce-af70-a8d331e7c2b8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-ld25h\" (UID: \"b4951dbc-6cb4-40ce-af70-a8d331e7c2b8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:12.397387 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.397255 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crv8\" (UniqueName: \"kubernetes.io/projected/7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca-kube-api-access-7crv8\") pod \"payload-processing-5f78687d5b-bvptt\" (UID: \"7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:12.397515 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.397388 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca-plugins-config-volume\") pod \"payload-processing-5f78687d5b-bvptt\" (UID: \"7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:12.398023 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.398000 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4951dbc-6cb4-40ce-af70-a8d331e7c2b8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-ld25h\" (UID: \"b4951dbc-6cb4-40ce-af70-a8d331e7c2b8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:12.420400 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.420340 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrtt\" (UniqueName: \"kubernetes.io/projected/b4951dbc-6cb4-40ce-af70-a8d331e7c2b8-kube-api-access-5wrtt\") pod \"payload-pre-processing-7b98bb678c-ld25h\" (UID: \"b4951dbc-6cb4-40ce-af70-a8d331e7c2b8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:12.498226 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.498174 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crv8\" (UniqueName: \"kubernetes.io/projected/7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca-kube-api-access-7crv8\") pod \"payload-processing-5f78687d5b-bvptt\" (UID: \"7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:12.498423 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.498231 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca-plugins-config-volume\") pod \"payload-processing-5f78687d5b-bvptt\" (UID: \"7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:12.499076 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.499050 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca-plugins-config-volume\") pod \"payload-processing-5f78687d5b-bvptt\" (UID: \"7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:12.521431 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.521409 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crv8\" (UniqueName: \"kubernetes.io/projected/7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca-kube-api-access-7crv8\") pod \"payload-processing-5f78687d5b-bvptt\" (UID: \"7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:12.573040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.573009 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:12.666428 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.666401 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:12.867486 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.867465 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-ld25h"] Oct 30 12:51:12.869471 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:51:12.869444 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4951dbc_6cb4_40ce_af70_a8d331e7c2b8.slice/crio-1f12793ee0c9ece05d6b80036a9284db19ace08fe442d6fa6c92eceb659890e3 WatchSource:0}: Error finding container 1f12793ee0c9ece05d6b80036a9284db19ace08fe442d6fa6c92eceb659890e3: Status 404 returned error can't find the container with id 1f12793ee0c9ece05d6b80036a9284db19ace08fe442d6fa6c92eceb659890e3 Oct 30 12:51:12.957645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:12.957619 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-bvptt"] Oct 30 12:51:12.959988 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:51:12.959962 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8cc90f_fd5b_4b71_ab60_5c2af7ee35ca.slice/crio-789cfa84db9e0b34a47a33f3bd0329158936f35af9c2feaa6742b97c317db715 WatchSource:0}: Error finding container 789cfa84db9e0b34a47a33f3bd0329158936f35af9c2feaa6742b97c317db715: Status 404 returned error can't find the container with id 789cfa84db9e0b34a47a33f3bd0329158936f35af9c2feaa6742b97c317db715 Oct 30 12:51:13.497037 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:13.496995 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" event={"ID":"7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca","Type":"ContainerStarted","Data":"789cfa84db9e0b34a47a33f3bd0329158936f35af9c2feaa6742b97c317db715"} Oct 30 12:51:13.499032 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:13.499007 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" event={"ID":"b4951dbc-6cb4-40ce-af70-a8d331e7c2b8","Type":"ContainerStarted","Data":"1f12793ee0c9ece05d6b80036a9284db19ace08fe442d6fa6c92eceb659890e3"} Oct 30 12:51:15.515459 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:15.515419 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" event={"ID":"7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca","Type":"ContainerStarted","Data":"4f60e19659de02c56f72b0034b6bafa541dace095c99a17fdc9cbf126fd80622"} Oct 30 12:51:15.515896 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:15.515505 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:15.517036 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:15.517012 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" event={"ID":"b4951dbc-6cb4-40ce-af70-a8d331e7c2b8","Type":"ContainerStarted","Data":"03f757993e2b6d2f6519e3dc916794c64d325a137a09e3b44e00c94d51410b59"} Oct 30 12:51:15.517157 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:15.517141 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:15.556032 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:15.555991 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" podStartSLOduration=1.391467108 podStartE2EDuration="3.555978763s" totalImagesPullingTime="2.164511655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:15.55512192 +0000 UTC m=+1093.681922817" watchObservedRunningTime="2026-10-30 12:51:15.555978763 +0000 UTC m=+1093.682779664" Oct 30 12:51:15.595102 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:15.595062 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" podStartSLOduration=1.426578048 podStartE2EDuration="3.595051246s" totalImagesPullingTime="2.168473198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:15.594044546 +0000 UTC m=+1093.720845441" watchObservedRunningTime="2026-10-30 12:51:15.595051246 +0000 UTC m=+1093.721852143" Oct 30 12:51:20.620229 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.620198 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z"] Oct 30 12:51:20.623663 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.623643 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.656652 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.656614 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z"] Oct 30 12:51:20.768196 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.768161 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.768402 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.768207 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.768402 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.768229 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.768402 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.768373 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.768518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.768406 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78162185-1d66-44e4-8652-06a94353866a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.768518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.768490 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78162185-1d66-44e4-8652-06a94353866a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.768518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.768515 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.768631 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.768546 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.768631 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.768599 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78h2\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-kube-api-access-m78h2\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.869611 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.869564 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.869611 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.869610 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.869823 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.869630 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.869823 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.869648 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78162185-1d66-44e4-8652-06a94353866a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.869823 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.869683 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78162185-1d66-44e4-8652-06a94353866a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.869823 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.869701 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.869823 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.869730 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.869823 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.869751 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78h2\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-kube-api-access-m78h2\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.870194 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.869965 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.870194 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.870047 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.870194 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.870084 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.870194 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.870106 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.870479 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.870355 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.870601 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.870579 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78162185-1d66-44e4-8652-06a94353866a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.872126 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.872100 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.872126 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.872119 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78162185-1d66-44e4-8652-06a94353866a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.893586 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.893558 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.893689 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.893673 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78h2\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-kube-api-access-m78h2\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rd49z\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:20.936188 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:20.936167 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:21.247246 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:21.243039 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z"] Oct 30 12:51:21.248444 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:21.248361 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:51:21.248708 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:21.248681 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:51:21.248898 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:21.248874 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:51:21.545345 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:21.545251 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" event={"ID":"78162185-1d66-44e4-8652-06a94353866a","Type":"ContainerStarted","Data":"97bd4bb657f18a83420b7d4502e837cb8b7a5a7fedfa7378a5f40cc08a841555"} Oct 30 12:51:21.545345 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:21.545323 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" event={"ID":"78162185-1d66-44e4-8652-06a94353866a","Type":"ContainerStarted","Data":"180884a0567d8218c47331a41db1848e7e2453d8f4dd5c156bfca3663a21fda1"} Oct 30 12:51:21.616284 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:21.616226 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" podStartSLOduration=1.6162115350000001 podStartE2EDuration="1.616211535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:21.60364174 +0000 UTC m=+1099.730442638" watchObservedRunningTime="2026-10-30 12:51:21.616211535 +0000 UTC m=+1099.743012433" Oct 30 12:51:21.936431 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:21.936380 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:22.489937 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.489909 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:22.816563 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.816476 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-8489fb66bd-4jwwm"] Oct 30 12:51:22.820868 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.820844 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:22.841297 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.841244 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8489fb66bd-4jwwm"] Oct 30 12:51:22.887681 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.887651 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1dcad86-b55c-40ed-9280-a639b93d15d4-metrics-certs\") pod \"ai-gateway-controller-8489fb66bd-4jwwm\" (UID: \"a1dcad86-b55c-40ed-9280-a639b93d15d4\") " pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:22.887836 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.887730 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgz6b\" (UniqueName: \"kubernetes.io/projected/a1dcad86-b55c-40ed-9280-a639b93d15d4-kube-api-access-zgz6b\") pod \"ai-gateway-controller-8489fb66bd-4jwwm\" (UID: \"a1dcad86-b55c-40ed-9280-a639b93d15d4\") " pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:22.938329 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.938293 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rd49z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:51:22.938702 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.938344 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:51:22.988631 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.988607 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1dcad86-b55c-40ed-9280-a639b93d15d4-metrics-certs\") pod \"ai-gateway-controller-8489fb66bd-4jwwm\" (UID: \"a1dcad86-b55c-40ed-9280-a639b93d15d4\") " pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:22.988734 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.988670 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgz6b\" (UniqueName: \"kubernetes.io/projected/a1dcad86-b55c-40ed-9280-a639b93d15d4-kube-api-access-zgz6b\") pod \"ai-gateway-controller-8489fb66bd-4jwwm\" (UID: \"a1dcad86-b55c-40ed-9280-a639b93d15d4\") " pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:22.990946 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:22.990922 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1dcad86-b55c-40ed-9280-a639b93d15d4-metrics-certs\") pod \"ai-gateway-controller-8489fb66bd-4jwwm\" (UID: \"a1dcad86-b55c-40ed-9280-a639b93d15d4\") " pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:23.020403 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:23.020378 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgz6b\" (UniqueName: \"kubernetes.io/projected/a1dcad86-b55c-40ed-9280-a639b93d15d4-kube-api-access-zgz6b\") pod \"ai-gateway-controller-8489fb66bd-4jwwm\" (UID: \"a1dcad86-b55c-40ed-9280-a639b93d15d4\") " pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:23.133484 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:23.133454 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:23.436740 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:23.436577 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8489fb66bd-4jwwm"] Oct 30 12:51:23.438871 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:51:23.438846 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1dcad86_b55c_40ed_9280_a639b93d15d4.slice/crio-ff11c4389dfffbe1ba30b3606dc9997d2701b5e0ec85cbe189ef78324a46eb0d WatchSource:0}: Error finding container ff11c4389dfffbe1ba30b3606dc9997d2701b5e0ec85cbe189ef78324a46eb0d: Status 404 returned error can't find the container with id ff11c4389dfffbe1ba30b3606dc9997d2701b5e0ec85cbe189ef78324a46eb0d Oct 30 12:51:23.555722 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:23.555686 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" event={"ID":"a1dcad86-b55c-40ed-9280-a639b93d15d4","Type":"ContainerStarted","Data":"ff11c4389dfffbe1ba30b3606dc9997d2701b5e0ec85cbe189ef78324a46eb0d"} Oct 30 12:51:23.937158 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:23.937124 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rd49z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:51:23.937327 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:23.937171 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:51:24.561064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:24.561032 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" event={"ID":"a1dcad86-b55c-40ed-9280-a639b93d15d4","Type":"ContainerStarted","Data":"6de08e01538baae3d130219bbf6e3dccf2f751b92db2b722fb10b3afa5ff6451"} Oct 30 12:51:24.561513 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:24.561103 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:24.617756 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:24.617690 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" podStartSLOduration=2.230616215 podStartE2EDuration="2.61766989s" totalImagesPullingTime="387.053675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:24.615653549 +0000 UTC m=+1102.742454448" watchObservedRunningTime="2026-10-30 12:51:24.61766989 +0000 UTC m=+1102.744470790" Oct 30 12:51:24.937328 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:24.937292 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rd49z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:51:24.937499 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:24.937347 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:51:25.936654 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:25.936618 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rd49z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:51:25.937038 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:25.936671 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:51:26.524238 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:26.524210 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-bvptt" Oct 30 12:51:26.524423 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:26.524309 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ld25h" Oct 30 12:51:26.936815 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:26.936780 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rd49z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 12:51:26.937195 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:26.936835 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 12:51:27.940492 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:27.940460 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:27.940984 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:27.940694 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:27.941484 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:27.941465 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 12:51:28.986825 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:51:28.986794 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:51:35.567838 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:35.567808 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-8489fb66bd-4jwwm" Oct 30 12:51:35.640533 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:35.640504 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ccqqn"] Oct 30 12:51:35.640772 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:35.640744 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" podUID="0ecba75f-13f3-46d6-8d7a-8493e0a37c54" containerName="manager" containerID="cri-o://28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26" gracePeriod=30 Oct 30 12:51:35.885097 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:35.885075 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:35.925254 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:35.925213 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-kube-api-access-pnxlx\" (UniqueName: \"kubernetes.io/projected/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-kube-api-access-pnxlx\") pod \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\" (UID: \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\") " Oct 30 12:51:35.925254 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:35.925258 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-metrics-certs\") pod \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\" (UID: \"0ecba75f-13f3-46d6-8d7a-8493e0a37c54\") " Oct 30 12:51:35.928160 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:35.928128 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-metrics-certs" pod "0ecba75f-13f3-46d6-8d7a-8493e0a37c54" (UID: "0ecba75f-13f3-46d6-8d7a-8493e0a37c54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:51:35.928160 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:35.928147 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-kube-api-access-pnxlx" pod "0ecba75f-13f3-46d6-8d7a-8493e0a37c54" (UID: "0ecba75f-13f3-46d6-8d7a-8493e0a37c54"). InnerVolumeSpecName "kube-api-access-pnxlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:51:36.026346 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.026315 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxlx\" (UniqueName: \"kubernetes.io/projected/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-kube-api-access-pnxlx\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:51:36.026346 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.026341 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecba75f-13f3-46d6-8d7a-8493e0a37c54-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:51:36.613932 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.613897 5515 generic.go:356] "Generic (PLEG): container finished" podID="0ecba75f-13f3-46d6-8d7a-8493e0a37c54" containerID="28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26" exitCode=0 Oct 30 12:51:36.614343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.613941 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" Oct 30 12:51:36.614343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.613975 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" event={"ID":"0ecba75f-13f3-46d6-8d7a-8493e0a37c54","Type":"ContainerDied","Data":"28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26"} Oct 30 12:51:36.614343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.614023 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ccqqn" event={"ID":"0ecba75f-13f3-46d6-8d7a-8493e0a37c54","Type":"ContainerDied","Data":"dceca723c981a62c396f9ebb0ecc17d3821caa0910f9204103e98ccbfa3f3b22"} Oct 30 12:51:36.614343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.614045 5515 scope.go:122] "RemoveContainer" containerID="28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26" Oct 30 12:51:36.622506 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.622486 5515 scope.go:122] "RemoveContainer" containerID="28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26" Oct 30 12:51:36.622784 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:51:36.622767 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26\": container with ID starting with 28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26 not found: ID does not exist" containerID="28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26" Oct 30 12:51:36.622835 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.622790 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26"} err="failed to get container status \"28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26\": rpc error: code = NotFound desc = could not find container \"28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26\": container with ID starting with 28669f66ea33ba3294abf4cf91f54fb36bfcaaec890b8f27c46ed8511e570f26 not found: ID does not exist" Oct 30 12:51:36.664642 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.664615 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ccqqn"] Oct 30 12:51:36.676411 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:36.676386 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ccqqn"] Oct 30 12:51:37.521115 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:37.521078 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:37.999195 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:37.999162 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ecba75f-13f3-46d6-8d7a-8493e0a37c54" path="/var/lib/kubelet/pods/0ecba75f-13f3-46d6-8d7a-8493e0a37c54/volumes" Oct 30 12:51:39.672081 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:39.672036 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:44.373042 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.373012 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h"] Oct 30 12:51:44.373551 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.373474 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0ecba75f-13f3-46d6-8d7a-8493e0a37c54" containerName="manager" Oct 30 12:51:44.373551 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.373488 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecba75f-13f3-46d6-8d7a-8493e0a37c54" containerName="manager" Oct 30 12:51:44.373682 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.373568 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecba75f-13f3-46d6-8d7a-8493e0a37c54" containerName="manager" Oct 30 12:51:44.377888 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.377869 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.404443 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 12:51:44.404983 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.404958 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2vklm\"" Oct 30 12:51:44.405146 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.405043 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 12:51:44.405146 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.405099 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 12:51:44.411291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.408833 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h"] Oct 30 12:51:44.505565 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.505533 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.505707 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.505563 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.505707 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.505589 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bac6b94-7947-41ce-aeb6-3daf8d972487-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.505707 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.505643 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.505828 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.505733 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.505828 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.505792 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x982t\" (UniqueName: \"kubernetes.io/projected/6bac6b94-7947-41ce-aeb6-3daf8d972487-kube-api-access-x982t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.517517 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.517490 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:44.606371 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.606336 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.606530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.606393 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x982t\" (UniqueName: \"kubernetes.io/projected/6bac6b94-7947-41ce-aeb6-3daf8d972487-kube-api-access-x982t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.606530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.606420 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.606530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.606434 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.606530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.606450 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bac6b94-7947-41ce-aeb6-3daf8d972487-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.606530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.606466 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.606825 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.606800 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.606932 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.606846 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.606981 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.606926 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.608720 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.608699 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bac6b94-7947-41ce-aeb6-3daf8d972487-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.608882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.608865 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bac6b94-7947-41ce-aeb6-3daf8d972487-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.629583 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.629533 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x982t\" (UniqueName: \"kubernetes.io/projected/6bac6b94-7947-41ce-aeb6-3daf8d972487-kube-api-access-x982t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h\" (UID: \"6bac6b94-7947-41ce-aeb6-3daf8d972487\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.690402 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.690377 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:44.989107 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:44.988927 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h"] Oct 30 12:51:44.991794 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:51:44.991767 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bac6b94_7947_41ce_aeb6_3daf8d972487.slice/crio-a0b5404e6971d954286978c508b7046e3304b02b5e132ca314d9191dfd7d4ba5 WatchSource:0}: Error finding container a0b5404e6971d954286978c508b7046e3304b02b5e132ca314d9191dfd7d4ba5: Status 404 returned error can't find the container with id a0b5404e6971d954286978c508b7046e3304b02b5e132ca314d9191dfd7d4ba5 Oct 30 12:51:45.546300 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:45.546250 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:51:45.656571 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:45.656507 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" event={"ID":"6bac6b94-7947-41ce-aeb6-3daf8d972487","Type":"ContainerStarted","Data":"a0b5404e6971d954286978c508b7046e3304b02b5e132ca314d9191dfd7d4ba5"} Oct 30 12:51:46.662142 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:46.662100 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" event={"ID":"6bac6b94-7947-41ce-aeb6-3daf8d972487","Type":"ContainerStarted","Data":"427deab578460f787e1cdb44ce5db85d08e917565ef5abff24845484015d6a0f"} Oct 30 12:51:46.691351 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:46.691326 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:46.718882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:46.718837 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" podStartSLOduration=1.3616703239999999 podStartE2EDuration="2.718825481s" totalImagesPullingTime="1.357155157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:46.717698658 +0000 UTC m=+1124.844499556" watchObservedRunningTime="2026-10-30 12:51:46.718825481 +0000 UTC m=+1124.845626357" Oct 30 12:51:51.022694 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.022638 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq"] Oct 30 12:51:51.026318 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.026294 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.042563 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.042538 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 12:51:51.055516 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.055478 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq"] Oct 30 12:51:51.168451 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.168409 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ea78e5-dd16-43b8-a556-4fea83e22c64-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.168451 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.168454 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.168680 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.168491 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.168680 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.168626 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.168745 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.168681 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zwnk\" (UniqueName: \"kubernetes.io/projected/b2ea78e5-dd16-43b8-a556-4fea83e22c64-kube-api-access-9zwnk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.168778 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.168756 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.187461 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.187432 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:51.269518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.269475 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.269698 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.269537 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.269698 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.269582 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.269698 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.269617 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zwnk\" (UniqueName: \"kubernetes.io/projected/b2ea78e5-dd16-43b8-a556-4fea83e22c64-kube-api-access-9zwnk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.269698 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.269671 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.269907 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.269732 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ea78e5-dd16-43b8-a556-4fea83e22c64-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.270063 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.270037 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.270125 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.270062 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.270125 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.270109 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.271855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.271838 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ea78e5-dd16-43b8-a556-4fea83e22c64-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.272143 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.272122 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ea78e5-dd16-43b8-a556-4fea83e22c64-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.297260 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.297196 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zwnk\" (UniqueName: \"kubernetes.io/projected/b2ea78e5-dd16-43b8-a556-4fea83e22c64-kube-api-access-9zwnk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pnkfq\" (UID: \"b2ea78e5-dd16-43b8-a556-4fea83e22c64\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.339715 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.339687 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:51.633288 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.633251 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq"] Oct 30 12:51:51.634741 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:51:51.634716 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ea78e5_dd16_43b8_a556_4fea83e22c64.slice/crio-650c15f425fcea38bbfe5f09334d5c4332f5628c97c1ed7afbeeb0b6a969206b WatchSource:0}: Error finding container 650c15f425fcea38bbfe5f09334d5c4332f5628c97c1ed7afbeeb0b6a969206b: Status 404 returned error can't find the container with id 650c15f425fcea38bbfe5f09334d5c4332f5628c97c1ed7afbeeb0b6a969206b Oct 30 12:51:51.685856 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:51.685822 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" event={"ID":"b2ea78e5-dd16-43b8-a556-4fea83e22c64","Type":"ContainerStarted","Data":"650c15f425fcea38bbfe5f09334d5c4332f5628c97c1ed7afbeeb0b6a969206b"} Oct 30 12:51:52.278872 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:52.278788 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:51:52.694835 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:52.694794 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" event={"ID":"b2ea78e5-dd16-43b8-a556-4fea83e22c64","Type":"ContainerStarted","Data":"ae4d5f58ccceaf3bf565308cf3c403b8f6423fc43b38dffce707d8d347e396e7"} Oct 30 12:51:53.340085 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:53.340041 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:51:54.423156 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:54.423119 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:54.691467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:54.691391 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:54.704747 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:54.704638 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:54.715997 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:54.715965 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" Oct 30 12:51:54.748822 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:54.748768 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" podStartSLOduration=4.531873821 podStartE2EDuration="4.748753451s" totalImagesPullingTime="216.87963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:52.741896356 +0000 UTC m=+1130.868697260" watchObservedRunningTime="2026-10-30 12:51:54.748753451 +0000 UTC m=+1132.875554352" Oct 30 12:51:57.046454 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:57.046415 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:57.861683 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:57.861649 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:51:58.953319 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:58.953289 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9"] Oct 30 12:51:58.966315 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:58.966289 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:58.978297 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:58.978248 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9"] Oct 30 12:51:58.988195 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:58.988166 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 12:51:59.000700 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:51:59.000510 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:51:59.053545 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.053504 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.053860 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.053822 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d17f60-c53a-4058-a144-88472194f6e5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.054068 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.054032 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.054310 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.054291 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.054468 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.054450 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.054654 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.054622 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhqv\" (UniqueName: \"kubernetes.io/projected/d7d17f60-c53a-4058-a144-88472194f6e5-kube-api-access-wrhqv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.116803 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.116760 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:59.161314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.155513 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhqv\" (UniqueName: \"kubernetes.io/projected/d7d17f60-c53a-4058-a144-88472194f6e5-kube-api-access-wrhqv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.161314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.155583 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.161314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.155647 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d17f60-c53a-4058-a144-88472194f6e5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.161314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.155697 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.161314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.155813 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.161314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.155841 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.161314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.157312 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.161314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.157577 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.161314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.157798 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.162943 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.162916 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d17f60-c53a-4058-a144-88472194f6e5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.163388 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.163364 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d17f60-c53a-4058-a144-88472194f6e5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.179665 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.179639 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhqv\" (UniqueName: \"kubernetes.io/projected/d7d17f60-c53a-4058-a144-88472194f6e5-kube-api-access-wrhqv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9\" (UID: \"d7d17f60-c53a-4058-a144-88472194f6e5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.288497 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.288406 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:51:59.595643 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.595621 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9"] Oct 30 12:51:59.597959 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:51:59.597933 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d17f60_c53a_4058_a144_88472194f6e5.slice/crio-53ec3230d374e715fdc2ec0819fce76ecd613fb2c6d19dc0253b664e329f662b WatchSource:0}: Error finding container 53ec3230d374e715fdc2ec0819fce76ecd613fb2c6d19dc0253b664e329f662b: Status 404 returned error can't find the container with id 53ec3230d374e715fdc2ec0819fce76ecd613fb2c6d19dc0253b664e329f662b Oct 30 12:51:59.751485 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:51:59.751450 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" event={"ID":"d7d17f60-c53a-4058-a144-88472194f6e5","Type":"ContainerStarted","Data":"53ec3230d374e715fdc2ec0819fce76ecd613fb2c6d19dc0253b664e329f662b"} Oct 30 12:52:00.433841 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:00.433812 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:00.757860 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:00.757781 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" event={"ID":"d7d17f60-c53a-4058-a144-88472194f6e5","Type":"ContainerStarted","Data":"8a380a13b0edab80b36e08e381ea73a6111d3bb6c3368dd45cf952cef899eeb4"} Oct 30 12:52:00.799508 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:00.799454 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" podStartSLOduration=2.57332589 podStartE2EDuration="2.799441359s" totalImagesPullingTime="226.115469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:00.799264595 +0000 UTC m=+1138.926065484" watchObservedRunningTime="2026-10-30 12:52:00.799441359 +0000 UTC m=+1138.926242257" Oct 30 12:52:01.289603 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:01.289570 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:52:01.340870 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:01.340833 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:52:01.353818 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:01.353794 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:52:01.775879 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:01.775850 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" Oct 30 12:52:02.772217 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.772182 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2"] Oct 30 12:52:02.777131 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.777100 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.799585 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.799552 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 12:52:02.827132 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.827095 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2"] Oct 30 12:52:02.891213 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.891178 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.891433 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.891405 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba929e02-50ce-44cf-af12-21d91c229651-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.891517 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.891503 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.891568 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.891530 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lb4\" (UniqueName: \"kubernetes.io/projected/ba929e02-50ce-44cf-af12-21d91c229651-kube-api-access-t8lb4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.891568 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.891554 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.891642 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.891615 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.965694 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.965663 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:02.992905 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.992872 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.993053 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.992929 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba929e02-50ce-44cf-af12-21d91c229651-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.993053 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.992977 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.993053 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.993002 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lb4\" (UniqueName: \"kubernetes.io/projected/ba929e02-50ce-44cf-af12-21d91c229651-kube-api-access-t8lb4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.993053 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.993031 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.993210 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.993054 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.993396 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.993370 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.993494 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.993438 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.993554 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.993535 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.995332 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.995309 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba929e02-50ce-44cf-af12-21d91c229651-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:02.995510 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:02.995494 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba929e02-50ce-44cf-af12-21d91c229651-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:03.016383 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:03.016355 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lb4\" (UniqueName: \"kubernetes.io/projected/ba929e02-50ce-44cf-af12-21d91c229651-kube-api-access-t8lb4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2\" (UID: \"ba929e02-50ce-44cf-af12-21d91c229651\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:03.091502 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:03.091413 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:03.410988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:03.410960 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2"] Oct 30 12:52:03.412431 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:52:03.412407 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba929e02_50ce_44cf_af12_21d91c229651.slice/crio-b19a4ba9c0ec0dd7aedbcf2154489258d3940f23504917997b84578af4efc376 WatchSource:0}: Error finding container b19a4ba9c0ec0dd7aedbcf2154489258d3940f23504917997b84578af4efc376: Status 404 returned error can't find the container with id b19a4ba9c0ec0dd7aedbcf2154489258d3940f23504917997b84578af4efc376 Oct 30 12:52:03.776019 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:03.775983 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" event={"ID":"ba929e02-50ce-44cf-af12-21d91c229651","Type":"ContainerStarted","Data":"35c47809d3571216bd2a106ebb3b8179ea0c7f68d8d7c8ba3dc784416a318684"} Oct 30 12:52:03.776182 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:03.776049 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" event={"ID":"ba929e02-50ce-44cf-af12-21d91c229651","Type":"ContainerStarted","Data":"b19a4ba9c0ec0dd7aedbcf2154489258d3940f23504917997b84578af4efc376"} Oct 30 12:52:03.819490 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:03.819446 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" podStartSLOduration=1.605540595 podStartE2EDuration="1.819432239s" totalImagesPullingTime="213.891644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:03.818460119 +0000 UTC m=+1141.945261016" watchObservedRunningTime="2026-10-30 12:52:03.819432239 +0000 UTC m=+1141.946233137" Oct 30 12:52:04.092323 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.092213 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:04.160660 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.160616 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc"] Oct 30 12:52:04.165008 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.164984 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.184339 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.184311 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc"] Oct 30 12:52:04.194811 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.194782 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 12:52:04.303910 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.303873 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.304068 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.303931 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.304148 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.304072 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543202e9-3b15-46b2-9dad-b6d056c087b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.304148 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.304134 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.304230 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.304192 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.304230 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.304216 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzh6\" (UniqueName: \"kubernetes.io/projected/543202e9-3b15-46b2-9dad-b6d056c087b7-kube-api-access-klzh6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.313455 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.313426 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.404993 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.404956 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.405186 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.405008 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzh6\" (UniqueName: \"kubernetes.io/projected/543202e9-3b15-46b2-9dad-b6d056c087b7-kube-api-access-klzh6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.405186 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.405054 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.405335 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.405210 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.405335 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.405316 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543202e9-3b15-46b2-9dad-b6d056c087b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.405440 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.405367 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.405501 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.405434 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.405584 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.405559 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.405739 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.405701 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.407324 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.407301 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543202e9-3b15-46b2-9dad-b6d056c087b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.407720 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.407703 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543202e9-3b15-46b2-9dad-b6d056c087b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.428579 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.428545 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzh6\" (UniqueName: \"kubernetes.io/projected/543202e9-3b15-46b2-9dad-b6d056c087b7-kube-api-access-klzh6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc\" (UID: \"543202e9-3b15-46b2-9dad-b6d056c087b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.491365 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.491322 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:04.787437 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:04.787414 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc"] Oct 30 12:52:04.789072 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:52:04.789043 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod543202e9_3b15_46b2_9dad_b6d056c087b7.slice/crio-6bcb831e6764e9295dd06ae734b80571acf37cd112f43532aac3d2b70cc34f3a WatchSource:0}: Error finding container 6bcb831e6764e9295dd06ae734b80571acf37cd112f43532aac3d2b70cc34f3a: Status 404 returned error can't find the container with id 6bcb831e6764e9295dd06ae734b80571acf37cd112f43532aac3d2b70cc34f3a Oct 30 12:52:05.548412 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.548379 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:52:05.782623 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.782591 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w"] Oct 30 12:52:05.787955 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.787926 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:05.790336 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.790298 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" event={"ID":"543202e9-3b15-46b2-9dad-b6d056c087b7","Type":"ContainerStarted","Data":"e249dd941b1e284d29964d6f884b0a6008f83442c72a2e17c13e07f924495f3b"} Oct 30 12:52:05.790484 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.790346 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" event={"ID":"543202e9-3b15-46b2-9dad-b6d056c087b7","Type":"ContainerStarted","Data":"6bcb831e6764e9295dd06ae734b80571acf37cd112f43532aac3d2b70cc34f3a"} Oct 30 12:52:05.817704 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.817630 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w"] Oct 30 12:52:05.872997 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.872941 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podStartSLOduration=1.6370621380000001 podStartE2EDuration="1.872924224s" totalImagesPullingTime="235.862086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:05.871101789 +0000 UTC m=+1143.997902681" watchObservedRunningTime="2026-10-30 12:52:05.872924224 +0000 UTC m=+1143.999725123" Oct 30 12:52:05.919736 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.919700 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:05.919736 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.919743 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:05.919990 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.919763 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:05.919990 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.919877 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a2964-91e7-4566-8121-59a460c1aeb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:05.920068 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.919982 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:05.920068 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.920031 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:05.920162 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.920098 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8z56\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-kube-api-access-q8z56\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:05.920162 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.920143 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:05.920303 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:05.920262 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021021 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.020978 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021218 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021139 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021218 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021176 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021218 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021204 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021470 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021237 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a2964-91e7-4566-8121-59a460c1aeb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021470 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021315 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021470 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021355 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021470 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021382 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8z56\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-kube-api-access-q8z56\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021470 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021412 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021730 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021602 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021793 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021744 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.021893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.021872 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.022580 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.022553 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a2964-91e7-4566-8121-59a460c1aeb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.022690 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.022645 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.024537 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.024503 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.024794 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.024769 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.046059 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.046025 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.046223 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.046200 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8z56\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-kube-api-access-q8z56\") pod \"maas-default-gateway-openshift-default-57f78bcd55-t696w\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.104167 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.104066 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:06.438729 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.438704 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w"] Oct 30 12:52:06.442005 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:52:06.441977 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6a2964_91e7_4566_8121_59a460c1aeb7.slice/crio-639beab3f171fdcd0a161d4f8841f9ed70f9b8ef8fa2fe9b845632efd8dcf4cf WatchSource:0}: Error finding container 639beab3f171fdcd0a161d4f8841f9ed70f9b8ef8fa2fe9b845632efd8dcf4cf: Status 404 returned error can't find the container with id 639beab3f171fdcd0a161d4f8841f9ed70f9b8ef8fa2fe9b845632efd8dcf4cf Oct 30 12:52:06.444368 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.444332 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:52:06.444478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.444398 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:52:06.444478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.444434 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:52:06.492356 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.492326 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:06.799827 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.799729 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" event={"ID":"1f6a2964-91e7-4566-8121-59a460c1aeb7","Type":"ContainerStarted","Data":"39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c"} Oct 30 12:52:06.799827 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.799801 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" event={"ID":"1f6a2964-91e7-4566-8121-59a460c1aeb7","Type":"ContainerStarted","Data":"639beab3f171fdcd0a161d4f8841f9ed70f9b8ef8fa2fe9b845632efd8dcf4cf"} Oct 30 12:52:06.850119 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:06.850066 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podStartSLOduration=1.8500521669999999 podStartE2EDuration="1.850052167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:06.848946947 +0000 UTC m=+1144.975747846" watchObservedRunningTime="2026-10-30 12:52:06.850052167 +0000 UTC m=+1144.976853064" Oct 30 12:52:07.107816 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:07.107679 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:07.113417 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:07.113031 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 12:52:07.113417 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:07.113111 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 12:52:07.862075 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:07.862041 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:08.105303 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:08.104687 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 12:52:08.105303 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:08.104748 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 12:52:09.108756 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:09.108719 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 12:52:09.109408 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:09.109367 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 12:52:09.289586 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:09.289539 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:52:09.318781 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:09.318747 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:52:09.838383 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:09.838354 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9" Oct 30 12:52:10.109830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:10.109728 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 12:52:10.110444 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:10.110406 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 12:52:11.118674 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:11.118636 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 12:52:11.119082 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:11.118690 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 12:52:12.116295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:12.113716 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 12:52:12.116295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:12.113782 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 12:52:13.092564 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:13.092534 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:13.105186 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:13.105156 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 12:52:13.105386 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:13.105210 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 12:52:13.105449 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:13.105380 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:13.857807 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:13.857774 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" Oct 30 12:52:14.492302 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:14.492256 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:14.505563 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:14.505535 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:14.818539 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:14.818458 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:14.819018 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:14.818992 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:14.819967 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:14.819942 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 12:52:14.863460 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:14.863433 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:52:23.194516 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:23.194481 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:23.404638 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:23.404605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:24.832283 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:24.832250 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:25.065512 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:25.065481 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:28.987808 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:52:28.987771 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:52:54.996423 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:54.996340 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:55.208526 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.208481 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:55.416621 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.416577 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:55.631481 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.631451 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t"] Oct 30 12:52:55.635312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.635280 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.635649 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.635619 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:55.656709 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.656688 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cc59b5af-openshift-default-dockercfg-zjd5w\"" Oct 30 12:52:55.670575 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.670505 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t"] Oct 30 12:52:55.756286 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.745662 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8350e-fb75-4613-ba55-717b478af302-istiod-ca-cert\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.756286 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.745860 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-socket\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.766251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.764413 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-data\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.766251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.764469 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-certs\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.766251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.764518 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mml6\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-kube-api-access-4mml6\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.766251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.764568 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8350e-fb75-4613-ba55-717b478af302-istio-podinfo\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.766251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.764617 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-istio-token\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.766251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.764660 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-envoy\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.766251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.764706 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-credential-socket\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866157 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866124 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-data\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866337 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866168 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-certs\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866337 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866296 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mml6\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-kube-api-access-4mml6\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866440 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866420 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8350e-fb75-4613-ba55-717b478af302-istio-podinfo\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866495 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866479 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-istio-token\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866549 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866522 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-certs\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866549 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866528 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-envoy\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866662 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866571 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-data\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866662 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866598 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-credential-socket\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866766 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866673 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8350e-fb75-4613-ba55-717b478af302-istiod-ca-cert\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866766 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866720 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-socket\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.866912 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.866888 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-credential-socket\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.867082 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.867056 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-socket\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.867519 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.867492 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8350e-fb75-4613-ba55-717b478af302-istiod-ca-cert\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.868766 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.868746 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-envoy\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.868917 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.868900 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8350e-fb75-4613-ba55-717b478af302-istio-podinfo\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.872684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.872646 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:55.891929 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.891901 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-istio-token\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.892877 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.892860 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mml6\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-kube-api-access-4mml6\") pod \"e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:55.955120 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:55.955033 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:56.170490 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.170453 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:56.205105 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.205071 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz"] Oct 30 12:52:56.214522 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.210175 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.237111 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.237080 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz"] Oct 30 12:52:56.246621 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.246597 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fb8b5ba5-gw-openshift-default-dockercfg-59vqb\"" Oct 30 12:52:56.269608 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.269571 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-data\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.269777 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.269622 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8g5n\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-kube-api-access-f8g5n\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.269777 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.269737 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-token\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.269777 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.269769 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-certs\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.269908 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.269791 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-envoy\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.269946 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.269914 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b3d16b-57f6-4a07-9903-2a002a10761b-istiod-ca-cert\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.269985 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.269961 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-podinfo\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.270020 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.269994 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-credential-socket\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.270132 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.270112 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-socket\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.283141 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.283118 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t"] Oct 30 12:52:56.285086 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:52:56.285064 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d8350e_fb75_4613_ba55_717b478af302.slice/crio-9ede9e1f33e9d080cb49660b797a481db09cc2ba470c2e80ebdf12aa1cb4a26d WatchSource:0}: Error finding container 9ede9e1f33e9d080cb49660b797a481db09cc2ba470c2e80ebdf12aa1cb4a26d: Status 404 returned error can't find the container with id 9ede9e1f33e9d080cb49660b797a481db09cc2ba470c2e80ebdf12aa1cb4a26d Oct 30 12:52:56.287339 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.287260 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:52:56.287420 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.287375 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:52:56.287420 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.287407 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:52:56.371052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371023 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-socket\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371198 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371057 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-data\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371198 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371075 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8g5n\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-kube-api-access-f8g5n\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371355 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371194 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-token\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371355 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371255 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-certs\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371355 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371315 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-envoy\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371368 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b3d16b-57f6-4a07-9903-2a002a10761b-istiod-ca-cert\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371424 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-podinfo\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371478 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-credential-socket\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371506 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-data\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371739 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371562 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-certs\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371739 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371695 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-socket\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.371840 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.371823 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-credential-socket\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.372152 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.372128 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b3d16b-57f6-4a07-9903-2a002a10761b-istiod-ca-cert\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.373398 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.373377 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-envoy\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.373878 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.373862 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-podinfo\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.383076 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.383052 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:56.396131 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.396107 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-token\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.396328 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.396307 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8g5n\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-kube-api-access-f8g5n\") pod \"e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.529365 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.529262 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:56.593307 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.592822 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:56.838568 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.838545 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz"] Oct 30 12:52:56.840524 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:52:56.840498 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b3d16b_57f6_4a07_9903_2a002a10761b.slice/crio-788a5b8c0c433d2853894bc562ba2ea22a5b6b7c45bd4d2ab5700e54fa021d21 WatchSource:0}: Error finding container 788a5b8c0c433d2853894bc562ba2ea22a5b6b7c45bd4d2ab5700e54fa021d21: Status 404 returned error can't find the container with id 788a5b8c0c433d2853894bc562ba2ea22a5b6b7c45bd4d2ab5700e54fa021d21 Oct 30 12:52:56.842730 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.842700 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:52:56.842810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.842764 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:52:56.842810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:56.842791 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:52:57.032476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.032438 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:57.060130 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.060080 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" event={"ID":"d5d8350e-fb75-4613-ba55-717b478af302","Type":"ContainerStarted","Data":"afa5a7a4d9161b0fdaba18827684dff25d874cae22490d89c52745e145e94d36"} Oct 30 12:52:57.060327 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.060153 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" event={"ID":"d5d8350e-fb75-4613-ba55-717b478af302","Type":"ContainerStarted","Data":"9ede9e1f33e9d080cb49660b797a481db09cc2ba470c2e80ebdf12aa1cb4a26d"} Oct 30 12:52:57.061896 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.061870 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" event={"ID":"50b3d16b-57f6-4a07-9903-2a002a10761b","Type":"ContainerStarted","Data":"f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a"} Oct 30 12:52:57.062006 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.061912 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" event={"ID":"50b3d16b-57f6-4a07-9903-2a002a10761b","Type":"ContainerStarted","Data":"788a5b8c0c433d2853894bc562ba2ea22a5b6b7c45bd4d2ab5700e54fa021d21"} Oct 30 12:52:57.113140 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.113045 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" podStartSLOduration=2.113030512 podStartE2EDuration="2.113030512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:57.110696436 +0000 UTC m=+1195.237497334" watchObservedRunningTime="2026-10-30 12:52:57.113030512 +0000 UTC m=+1195.239831410" Oct 30 12:52:57.157567 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.157516 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" podStartSLOduration=1.157501756 podStartE2EDuration="1.157501756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:57.155840638 +0000 UTC m=+1195.282641572" watchObservedRunningTime="2026-10-30 12:52:57.157501756 +0000 UTC m=+1195.284302654" Oct 30 12:52:57.530331 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.529848 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:57.541989 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.541950 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:57.571332 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.567724 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:57.746462 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.746432 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:57.956065 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.956036 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:57.960571 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:57.960547 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:58.067884 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:58.067529 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:58.067884 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:58.067588 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:58.068373 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:58.068342 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:52:58.068500 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:58.068400 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:52:58.989023 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:52:58.988991 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:52:59.144898 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:59.144855 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:59.400480 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:59.400441 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:59.606822 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:59.606791 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:59.806164 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:52:59.806087 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:00.394497 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.394465 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:00.601306 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.601256 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:00.629900 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.629865 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6"] Oct 30 12:53:00.633398 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.633380 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:00.663216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.663138 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fb8b5ba5-dockercfg-nmx7b\"" Oct 30 12:53:00.663216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.663145 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fb8b5ba5\"" Oct 30 12:53:00.665137 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.665117 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6"] Oct 30 12:53:00.727151 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.727107 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffdnx\" (UniqueName: \"kubernetes.io/projected/ecc31286-ddda-485c-9581-28c31e0fd3b3-kube-api-access-ffdnx\") pod \"payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6\" (UID: \"ecc31286-ddda-485c-9581-28c31e0fd3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:00.727424 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.727306 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc31286-ddda-485c-9581-28c31e0fd3b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6\" (UID: \"ecc31286-ddda-485c-9581-28c31e0fd3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:00.817343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.817311 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:00.827974 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.827947 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc31286-ddda-485c-9581-28c31e0fd3b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6\" (UID: \"ecc31286-ddda-485c-9581-28c31e0fd3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:00.828126 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.828057 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffdnx\" (UniqueName: \"kubernetes.io/projected/ecc31286-ddda-485c-9581-28c31e0fd3b3-kube-api-access-ffdnx\") pod \"payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6\" (UID: \"ecc31286-ddda-485c-9581-28c31e0fd3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:00.828682 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.828664 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc31286-ddda-485c-9581-28c31e0fd3b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6\" (UID: \"ecc31286-ddda-485c-9581-28c31e0fd3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:00.851444 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.851418 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffdnx\" (UniqueName: \"kubernetes.io/projected/ecc31286-ddda-485c-9581-28c31e0fd3b3-kube-api-access-ffdnx\") pod \"payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6\" (UID: \"ecc31286-ddda-485c-9581-28c31e0fd3b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:00.946932 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:00.946847 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:01.252296 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:01.252248 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6"] Oct 30 12:53:01.253713 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:01.253687 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc31286_ddda_485c_9581_28c31e0fd3b3.slice/crio-5c45fd312b8529743385f3b4aba89ae71449c86196f5812d6aa83b99dc052ddd WatchSource:0}: Error finding container 5c45fd312b8529743385f3b4aba89ae71449c86196f5812d6aa83b99dc052ddd: Status 404 returned error can't find the container with id 5c45fd312b8529743385f3b4aba89ae71449c86196f5812d6aa83b99dc052ddd Oct 30 12:53:02.089076 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.089033 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" event={"ID":"ecc31286-ddda-485c-9581-28c31e0fd3b3","Type":"ContainerStarted","Data":"1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267"} Oct 30 12:53:02.089601 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.089106 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:02.089601 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.089133 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" event={"ID":"ecc31286-ddda-485c-9581-28c31e0fd3b3","Type":"ContainerStarted","Data":"5c45fd312b8529743385f3b4aba89ae71449c86196f5812d6aa83b99dc052ddd"} Oct 30 12:53:02.134506 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.134459 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" podStartSLOduration=2.13444559 podStartE2EDuration="2.13444559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:02.132745522 +0000 UTC m=+1200.259546491" watchObservedRunningTime="2026-10-30 12:53:02.13444559 +0000 UTC m=+1200.261246518" Oct 30 12:53:02.339996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.339920 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:02.479197 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.479163 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt"] Oct 30 12:53:02.482935 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.482918 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:02.500717 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.500687 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cc59b5af-dockercfg-4pqmc\"" Oct 30 12:53:02.503562 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.503537 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cc59b5af\"" Oct 30 12:53:02.505633 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.505611 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt"] Oct 30 12:53:02.554886 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.554842 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03676fec-d216-480d-917e-6d084b220599-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt\" (UID: \"03676fec-d216-480d-917e-6d084b220599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:02.555062 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.554964 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvdg\" (UniqueName: \"kubernetes.io/projected/03676fec-d216-480d-917e-6d084b220599-kube-api-access-swvdg\") pod \"payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt\" (UID: \"03676fec-d216-480d-917e-6d084b220599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:02.556415 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.556386 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:02.577318 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.577255 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz"] Oct 30 12:53:02.593673 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.593596 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:02.601228 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.600832 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz"] Oct 30 12:53:02.656237 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.656202 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03676fec-d216-480d-917e-6d084b220599-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt\" (UID: \"03676fec-d216-480d-917e-6d084b220599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:02.656463 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.656305 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d440fd2-f9f6-479f-bb17-471ea645e4a8-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz\" (UID: \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\") " pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:02.656463 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.656331 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9cqq\" (UniqueName: \"kubernetes.io/projected/0d440fd2-f9f6-479f-bb17-471ea645e4a8-kube-api-access-h9cqq\") pod \"payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz\" (UID: \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\") " pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:02.656463 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.656355 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvdg\" (UniqueName: \"kubernetes.io/projected/03676fec-d216-480d-917e-6d084b220599-kube-api-access-swvdg\") pod \"payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt\" (UID: \"03676fec-d216-480d-917e-6d084b220599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:02.656940 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.656919 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03676fec-d216-480d-917e-6d084b220599-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt\" (UID: \"03676fec-d216-480d-917e-6d084b220599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:02.685042 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.685013 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvdg\" (UniqueName: \"kubernetes.io/projected/03676fec-d216-480d-917e-6d084b220599-kube-api-access-swvdg\") pod \"payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt\" (UID: \"03676fec-d216-480d-917e-6d084b220599\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:02.757055 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.757023 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d440fd2-f9f6-479f-bb17-471ea645e4a8-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz\" (UID: \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\") " pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:02.757055 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.757058 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9cqq\" (UniqueName: \"kubernetes.io/projected/0d440fd2-f9f6-479f-bb17-471ea645e4a8-kube-api-access-h9cqq\") pod \"payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz\" (UID: \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\") " pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:02.757799 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.757774 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d440fd2-f9f6-479f-bb17-471ea645e4a8-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz\" (UID: \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\") " pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:02.796441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.796406 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:02.808762 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.808733 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9cqq\" (UniqueName: \"kubernetes.io/projected/0d440fd2-f9f6-479f-bb17-471ea645e4a8-kube-api-access-h9cqq\") pod \"payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz\" (UID: \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\") " pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:02.914416 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:02.914390 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:03.097711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.097654 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt"] Oct 30 12:53:03.098394 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:03.098374 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03676fec_d216_480d_917e_6d084b220599.slice/crio-8bd7871c9e1d30d99e30c10a1dc8023faceba8b4c316820f84c0b5c4280fc47f WatchSource:0}: Error finding container 8bd7871c9e1d30d99e30c10a1dc8023faceba8b4c316820f84c0b5c4280fc47f: Status 404 returned error can't find the container with id 8bd7871c9e1d30d99e30c10a1dc8023faceba8b4c316820f84c0b5c4280fc47f Oct 30 12:53:03.214335 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.214223 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz"] Oct 30 12:53:03.223879 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:03.223840 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d440fd2_f9f6_479f_bb17_471ea645e4a8.slice/crio-314cdab3cc0a77a035b0824ae305dab0fdb0488fc6cb67e45e9d7a00fcc0bac5 WatchSource:0}: Error finding container 314cdab3cc0a77a035b0824ae305dab0fdb0488fc6cb67e45e9d7a00fcc0bac5: Status 404 returned error can't find the container with id 314cdab3cc0a77a035b0824ae305dab0fdb0488fc6cb67e45e9d7a00fcc0bac5 Oct 30 12:53:03.378343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.378234 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:03.559949 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.559912 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j"] Oct 30 12:53:03.567555 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.567529 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:03.593448 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.591175 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j"] Oct 30 12:53:03.593824 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.593792 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:03.600372 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.600345 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f1c4045f\"" Oct 30 12:53:03.600501 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.600426 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f1c4045f-dockercfg-f4bp2\"" Oct 30 12:53:03.666503 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.666457 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-plugins-config-volume\") pod \"payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j\" (UID: \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\") " pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:03.666699 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.666579 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlnm\" (UniqueName: \"kubernetes.io/projected/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-kube-api-access-6vlnm\") pod \"payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j\" (UID: \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\") " pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:03.767459 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.767423 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlnm\" (UniqueName: \"kubernetes.io/projected/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-kube-api-access-6vlnm\") pod \"payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j\" (UID: \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\") " pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:03.767644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.767559 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-plugins-config-volume\") pod \"payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j\" (UID: \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\") " pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:03.768318 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.768298 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-plugins-config-volume\") pod \"payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j\" (UID: \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\") " pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:03.793202 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.793163 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlnm\" (UniqueName: \"kubernetes.io/projected/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-kube-api-access-6vlnm\") pod \"payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j\" (UID: \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\") " pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:03.836704 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.836453 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:03.901901 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:03.901867 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:04.060669 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.060636 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:04.089833 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.089805 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp"] Oct 30 12:53:04.094433 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.094413 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.102860 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.102828 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" event={"ID":"03676fec-d216-480d-917e-6d084b220599","Type":"ContainerStarted","Data":"4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb"} Oct 30 12:53:04.103232 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.102873 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" event={"ID":"03676fec-d216-480d-917e-6d084b220599","Type":"ContainerStarted","Data":"8bd7871c9e1d30d99e30c10a1dc8023faceba8b4c316820f84c0b5c4280fc47f"} Oct 30 12:53:04.103232 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.102984 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:04.104456 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.104431 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" event={"ID":"0d440fd2-f9f6-479f-bb17-471ea645e4a8","Type":"ContainerStarted","Data":"b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7"} Oct 30 12:53:04.104547 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.104467 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" event={"ID":"0d440fd2-f9f6-479f-bb17-471ea645e4a8","Type":"ContainerStarted","Data":"314cdab3cc0a77a035b0824ae305dab0fdb0488fc6cb67e45e9d7a00fcc0bac5"} Oct 30 12:53:04.104547 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.104494 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:04.111693 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.111669 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fb8b5ba5-serving-cert\"" Oct 30 12:53:04.111885 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.111869 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dlqhq\"" Oct 30 12:53:04.116804 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.116749 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 12:53:04.120798 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.119137 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp"] Oct 30 12:53:04.172248 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.172218 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-metrics-certs\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.172440 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.172345 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-maas-api-tls\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.172440 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.172380 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c9sc\" (UniqueName: \"kubernetes.io/projected/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-kube-api-access-9c9sc\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.172667 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.172650 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-openshift-service-ca\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.212789 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.212761 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j"] Oct 30 12:53:04.212916 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.212817 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" podStartSLOduration=2.212804769 podStartE2EDuration="2.212804769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:04.209787581 +0000 UTC m=+1202.336588478" watchObservedRunningTime="2026-10-30 12:53:04.212804769 +0000 UTC m=+1202.339605669" Oct 30 12:53:04.213905 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:04.213878 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d31c1bf_de1b_4c72_8560_0dfe4a7bedf1.slice/crio-0fe1ac57ac10651ab7a8fc998a2cdfc7767ca26a4321ba0fddc02a8d9e585535 WatchSource:0}: Error finding container 0fe1ac57ac10651ab7a8fc998a2cdfc7767ca26a4321ba0fddc02a8d9e585535: Status 404 returned error can't find the container with id 0fe1ac57ac10651ab7a8fc998a2cdfc7767ca26a4321ba0fddc02a8d9e585535 Oct 30 12:53:04.252763 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.252716 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" podStartSLOduration=2.25270189 podStartE2EDuration="2.25270189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:04.251423551 +0000 UTC m=+1202.378224450" watchObservedRunningTime="2026-10-30 12:53:04.25270189 +0000 UTC m=+1202.379502788" Oct 30 12:53:04.268960 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.268903 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:04.274208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.274183 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-maas-api-tls\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.274338 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.274221 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c9sc\" (UniqueName: \"kubernetes.io/projected/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-kube-api-access-9c9sc\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.274338 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.274309 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-openshift-service-ca\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.274426 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.274401 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-metrics-certs\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.275135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.275108 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-openshift-service-ca\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.276483 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.276464 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-maas-api-tls\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.276564 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.276531 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-metrics-certs\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.297378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.297356 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c9sc\" (UniqueName: \"kubernetes.io/projected/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-kube-api-access-9c9sc\") pod \"maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.408848 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.408817 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:04.541464 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.541414 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:04.729506 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:04.729469 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7f0f9e_ec9b_4d96_a066_dab0a174a3e3.slice/crio-ed5c4ec09a8e1347324418266c4e50772dff5c42ed5364cf3ed0ce73fa6649a5 WatchSource:0}: Error finding container ed5c4ec09a8e1347324418266c4e50772dff5c42ed5364cf3ed0ce73fa6649a5: Status 404 returned error can't find the container with id ed5c4ec09a8e1347324418266c4e50772dff5c42ed5364cf3ed0ce73fa6649a5 Oct 30 12:53:04.736369 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.736251 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp"] Oct 30 12:53:04.756535 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:04.753788 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:05.008700 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:05.008620 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:05.110308 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:05.110261 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" event={"ID":"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3","Type":"ContainerStarted","Data":"ed5c4ec09a8e1347324418266c4e50772dff5c42ed5364cf3ed0ce73fa6649a5"} Oct 30 12:53:05.111717 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:05.111681 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" event={"ID":"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1","Type":"ContainerStarted","Data":"72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5"} Oct 30 12:53:05.112075 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:05.112060 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:05.112134 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:05.112101 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" event={"ID":"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1","Type":"ContainerStarted","Data":"0fe1ac57ac10651ab7a8fc998a2cdfc7767ca26a4321ba0fddc02a8d9e585535"} Oct 30 12:53:05.153660 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:05.153616 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" podStartSLOduration=2.153603096 podStartE2EDuration="2.153603096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:05.153319862 +0000 UTC m=+1203.280120761" watchObservedRunningTime="2026-10-30 12:53:05.153603096 +0000 UTC m=+1203.280404096" Oct 30 12:53:06.031029 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.030997 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:06.119347 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.119312 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" event={"ID":"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3","Type":"ContainerStarted","Data":"ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780"} Oct 30 12:53:06.119786 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.119415 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:06.202990 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.202288 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" podStartSLOduration=1.814990461 podStartE2EDuration="2.202255178s" totalImagesPullingTime="387.264717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:06.20169502 +0000 UTC m=+1204.328495931" watchObservedRunningTime="2026-10-30 12:53:06.202255178 +0000 UTC m=+1204.329056079" Oct 30 12:53:06.250212 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.250174 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9"] Oct 30 12:53:06.254079 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.254062 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:06.274516 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.274480 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9"] Oct 30 12:53:06.304017 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.303935 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:06.332348 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.332320 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-873e4f54\"" Oct 30 12:53:06.332348 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.332355 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-873e4f54-dockercfg-cqgcg\"" Oct 30 12:53:06.400799 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.400765 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7dj\" (UniqueName: \"kubernetes.io/projected/74990b01-7eaf-48bd-9a1f-12e3abb16556-kube-api-access-js7dj\") pod \"payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9\" (UID: \"74990b01-7eaf-48bd-9a1f-12e3abb16556\") " pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:06.400981 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.400828 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74990b01-7eaf-48bd-9a1f-12e3abb16556-plugins-config-volume\") pod \"payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9\" (UID: \"74990b01-7eaf-48bd-9a1f-12e3abb16556\") " pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:06.501667 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.501631 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7dj\" (UniqueName: \"kubernetes.io/projected/74990b01-7eaf-48bd-9a1f-12e3abb16556-kube-api-access-js7dj\") pod \"payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9\" (UID: \"74990b01-7eaf-48bd-9a1f-12e3abb16556\") " pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:06.501851 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.501727 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74990b01-7eaf-48bd-9a1f-12e3abb16556-plugins-config-volume\") pod \"payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9\" (UID: \"74990b01-7eaf-48bd-9a1f-12e3abb16556\") " pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:06.502514 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.502490 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74990b01-7eaf-48bd-9a1f-12e3abb16556-plugins-config-volume\") pod \"payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9\" (UID: \"74990b01-7eaf-48bd-9a1f-12e3abb16556\") " pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:06.524585 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.524551 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7dj\" (UniqueName: \"kubernetes.io/projected/74990b01-7eaf-48bd-9a1f-12e3abb16556-kube-api-access-js7dj\") pod \"payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9\" (UID: \"74990b01-7eaf-48bd-9a1f-12e3abb16556\") " pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:06.567702 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.567628 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:06.681306 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.681182 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:53:06.878950 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:06.878920 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9"] Oct 30 12:53:06.880502 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:06.880472 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74990b01_7eaf_48bd_9a1f_12e3abb16556.slice/crio-ece2f246a11a61635674ddf8c8bf7571f786715a686642ab18548d4fffabdd59 WatchSource:0}: Error finding container ece2f246a11a61635674ddf8c8bf7571f786715a686642ab18548d4fffabdd59: Status 404 returned error can't find the container with id ece2f246a11a61635674ddf8c8bf7571f786715a686642ab18548d4fffabdd59 Oct 30 12:53:07.137381 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:07.136483 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" event={"ID":"74990b01-7eaf-48bd-9a1f-12e3abb16556","Type":"ContainerStarted","Data":"ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0"} Oct 30 12:53:07.137381 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:07.136542 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" event={"ID":"74990b01-7eaf-48bd-9a1f-12e3abb16556","Type":"ContainerStarted","Data":"ece2f246a11a61635674ddf8c8bf7571f786715a686642ab18548d4fffabdd59"} Oct 30 12:53:07.137381 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:07.137157 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:07.177074 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:07.177021 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" podStartSLOduration=1.177007647 podStartE2EDuration="1.177007647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:07.176326639 +0000 UTC m=+1205.303127538" watchObservedRunningTime="2026-10-30 12:53:07.177007647 +0000 UTC m=+1205.303808545" Oct 30 12:53:07.651585 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:07.651548 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:07.865641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:07.865206 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:08.092203 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:08.092122 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:08.308291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:08.294523 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:09.592158 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:09.592129 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:09.824865 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:09.824789 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:09.920945 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:09.920905 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s"] Oct 30 12:53:09.924910 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:09.924891 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:09.944330 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:09.943835 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s"] Oct 30 12:53:10.039744 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.039712 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cc59b5af-serving-cert\"" Oct 30 12:53:10.062226 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.062193 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-openshift-service-ca\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.062379 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.062340 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-maas-api-tls\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.062420 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.062382 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-metrics-certs\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.062420 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.062409 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f48bn\" (UniqueName: \"kubernetes.io/projected/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-kube-api-access-f48bn\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.086324 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.086294 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:10.163839 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.163803 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-openshift-service-ca\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.163990 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.163931 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-maas-api-tls\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.164050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.163994 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-metrics-certs\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.164050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.164028 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f48bn\" (UniqueName: \"kubernetes.io/projected/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-kube-api-access-f48bn\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.164584 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.164556 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-openshift-service-ca\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.166461 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.166443 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-metrics-certs\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.166536 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.166497 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-maas-api-tls\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.186855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.186795 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f48bn\" (UniqueName: \"kubernetes.io/projected/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-kube-api-access-f48bn\") pod \"maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.238922 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.238888 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:10.292991 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.292342 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:10.486990 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.486915 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:10.561477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.561446 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s"] Oct 30 12:53:10.563221 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:10.563196 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6630deb8_bbdc_4b8c_88e0_da0fd2f6c31e.slice/crio-f71b94a1384cb5b13ec0a20fb34b25b9f8ba9b47e5a0062b199428c584d95fa5 WatchSource:0}: Error finding container f71b94a1384cb5b13ec0a20fb34b25b9f8ba9b47e5a0062b199428c584d95fa5: Status 404 returned error can't find the container with id f71b94a1384cb5b13ec0a20fb34b25b9f8ba9b47e5a0062b199428c584d95fa5 Oct 30 12:53:10.725690 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:10.725654 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:53:11.161653 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:11.161619 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" event={"ID":"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e","Type":"ContainerStarted","Data":"77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a"} Oct 30 12:53:11.161829 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:11.161667 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" event={"ID":"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e","Type":"ContainerStarted","Data":"f71b94a1384cb5b13ec0a20fb34b25b9f8ba9b47e5a0062b199428c584d95fa5"} Oct 30 12:53:11.161829 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:11.161741 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:11.990988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:11.990950 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:12.143088 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:12.143061 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:12.186570 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:12.186521 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" podStartSLOduration=2.8311333469999997 podStartE2EDuration="3.186508514s" totalImagesPullingTime="355.375167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:11.208028885 +0000 UTC m=+1209.334829786" watchObservedRunningTime="2026-10-30 12:53:12.186508514 +0000 UTC m=+1210.313309411" Oct 30 12:53:13.120412 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:13.120379 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:13.683467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:13.683434 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:15.122906 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:15.122875 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:15.123576 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:15.123545 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:15.487760 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:15.487722 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:15.688239 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:15.688200 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:17.139510 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:17.139483 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:17.172345 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:17.172318 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:17.228378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:17.228286 5515 generic.go:356] "Generic (PLEG): container finished" podID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerID="f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a" exitCode=0 Oct 30 12:53:17.228378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:17.228362 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" event={"ID":"50b3d16b-57f6-4a07-9903-2a002a10761b","Type":"ContainerDied","Data":"f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a"} Oct 30 12:53:17.228727 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:17.228712 5515 scope.go:122] "RemoveContainer" containerID="f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a" Oct 30 12:53:17.230078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:17.230043 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:17.230148 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:17.230133 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:17.230232 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:17.230212 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:17.335963 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:17.335932 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:18.247738 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:18.247695 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" event={"ID":"50b3d16b-57f6-4a07-9903-2a002a10761b","Type":"ContainerStarted","Data":"15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481"} Oct 30 12:53:18.538350 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:18.537701 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:53:18.538350 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:18.537831 5515 patch_prober.go:29] interesting pod/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:53:18.538350 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:18.537873 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 12:53:19.117301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:19.116831 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:19.148441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:19.148413 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:19.369344 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:19.361696 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:53:19.530972 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:19.529678 5515 patch_prober.go:29] interesting pod/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:53:19.530972 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:19.529762 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 12:53:20.530015 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:20.529978 5515 patch_prober.go:29] interesting pod/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:53:20.530429 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:20.530042 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 12:53:21.145758 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:21.145669 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:21.530333 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:21.530075 5515 patch_prober.go:29] interesting pod/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:53:21.530333 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:21.530133 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 12:53:22.530304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:22.530263 5515 patch_prober.go:29] interesting pod/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 12:53:22.530497 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:22.530325 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 12:53:23.083544 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:23.083516 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:23.637638 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:23.637600 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:53:23.638517 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:23.638488 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:53:23.640772 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:23.640737 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:53:24.286431 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:24.286351 5515 generic.go:356] "Generic (PLEG): container finished" podID="d5d8350e-fb75-4613-ba55-717b478af302" containerID="afa5a7a4d9161b0fdaba18827684dff25d874cae22490d89c52745e145e94d36" exitCode=0 Oct 30 12:53:24.286607 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:24.286434 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" event={"ID":"d5d8350e-fb75-4613-ba55-717b478af302","Type":"ContainerDied","Data":"afa5a7a4d9161b0fdaba18827684dff25d874cae22490d89c52745e145e94d36"} Oct 30 12:53:24.286852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:24.286838 5515 scope.go:122] "RemoveContainer" containerID="afa5a7a4d9161b0fdaba18827684dff25d874cae22490d89c52745e145e94d36" Oct 30 12:53:24.287957 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:24.287931 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:24.288017 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:24.288000 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:24.288053 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:24.288032 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:24.940808 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:24.940773 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:25.068134 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.068098 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz"] Oct 30 12:53:25.071051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.071028 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.085566 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.085546 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:53:25.115290 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.112529 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\"" Oct 30 12:53:25.123533 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.123500 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz"] Oct 30 12:53:25.250710 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.250616 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.250863 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.250808 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8d4\" (UniqueName: \"kubernetes.io/projected/e4f8450f-01fa-49cb-b68e-44ab467f5b02-kube-api-access-hs8d4\") pod \"maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.250863 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.250843 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.318489 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.318438 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" event={"ID":"d5d8350e-fb75-4613-ba55-717b478af302","Type":"ContainerStarted","Data":"31f5a875c9b4aef3d216d583d392ceda891657a0e973b6c370f8e03d8b6cdaac"} Oct 30 12:53:25.351624 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.351592 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8d4\" (UniqueName: \"kubernetes.io/projected/e4f8450f-01fa-49cb-b68e-44ab467f5b02-kube-api-access-hs8d4\") pod \"maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.351624 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.351629 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.351876 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.351690 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.352395 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.352373 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.353928 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.353906 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.374813 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.374791 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8d4\" (UniqueName: \"kubernetes.io/projected/e4f8450f-01fa-49cb-b68e-44ab467f5b02-kube-api-access-hs8d4\") pod \"maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.383677 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.383651 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:25.718818 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.718769 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz"] Oct 30 12:53:25.722300 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:25.722185 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f8450f_01fa_49cb_b68e_44ab467f5b02.slice/crio-03430ccb4a478aeb4b63bbcb245660b13f0b5da2f4bf2d6d973b8a705a8af031 WatchSource:0}: Error finding container 03430ccb4a478aeb4b63bbcb245660b13f0b5da2f4bf2d6d973b8a705a8af031: Status 404 returned error can't find the container with id 03430ccb4a478aeb4b63bbcb245660b13f0b5da2f4bf2d6d973b8a705a8af031 Oct 30 12:53:25.955498 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.955465 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:53:25.956733 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.956711 5515 patch_prober.go:29] interesting pod/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:53:25.956792 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:25.956757 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 12:53:26.326571 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:26.326535 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" event={"ID":"e4f8450f-01fa-49cb-b68e-44ab467f5b02","Type":"ContainerStarted","Data":"03430ccb4a478aeb4b63bbcb245660b13f0b5da2f4bf2d6d973b8a705a8af031"} Oct 30 12:53:26.519026 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:26.518993 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:26.956369 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:26.956331 5515 patch_prober.go:29] interesting pod/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:53:26.956799 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:26.956394 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 12:53:27.955897 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:27.955859 5515 patch_prober.go:29] interesting pod/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:53:27.956102 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:27.955925 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 12:53:28.068234 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:28.068205 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:53:28.371877 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:28.371839 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:28.956034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:28.956001 5515 patch_prober.go:29] interesting pod/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:53:28.956214 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:28.956054 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 12:53:28.987706 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:28.987674 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:53:29.354162 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:29.354074 5515 generic.go:356] "Generic (PLEG): container finished" podID="e4f8450f-01fa-49cb-b68e-44ab467f5b02" containerID="b52aa46b10124c59ccc41c420a03f3b388ed14d5fbd7ad176f413d8037ecaccb" exitCode=0 Oct 30 12:53:29.354539 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:29.354160 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" event={"ID":"e4f8450f-01fa-49cb-b68e-44ab467f5b02","Type":"ContainerDied","Data":"b52aa46b10124c59ccc41c420a03f3b388ed14d5fbd7ad176f413d8037ecaccb"} Oct 30 12:53:29.669534 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:29.669505 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6"] Oct 30 12:53:29.874102 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:29.874060 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:29.955736 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:29.955660 5515 patch_prober.go:29] interesting pod/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:53:29.955736 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:29.955715 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 12:53:30.078803 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.078753 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:30.544567 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.544539 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:30.635390 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.635366 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-openshift-ca\") pod \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " Oct 30 12:53:30.635545 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.635448 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-xks-ca\") pod \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " Oct 30 12:53:30.635545 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.635503 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f8450f-01fa-49cb-b68e-44ab467f5b02-kube-api-access-hs8d4\" (UniqueName: \"kubernetes.io/projected/e4f8450f-01fa-49cb-b68e-44ab467f5b02-kube-api-access-hs8d4\") pod \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\" (UID: \"e4f8450f-01fa-49cb-b68e-44ab467f5b02\") " Oct 30 12:53:30.635811 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.635788 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-openshift-ca" pod "e4f8450f-01fa-49cb-b68e-44ab467f5b02" (UID: "e4f8450f-01fa-49cb-b68e-44ab467f5b02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:30.635919 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.635894 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-openshift-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:30.637603 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.637569 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f8450f-01fa-49cb-b68e-44ab467f5b02-kube-api-access-hs8d4" pod "e4f8450f-01fa-49cb-b68e-44ab467f5b02" (UID: "e4f8450f-01fa-49cb-b68e-44ab467f5b02"). InnerVolumeSpecName "kube-api-access-hs8d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:30.637704 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.637629 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-xks-ca" pod "e4f8450f-01fa-49cb-b68e-44ab467f5b02" (UID: "e4f8450f-01fa-49cb-b68e-44ab467f5b02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:30.736864 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.736823 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f8450f-01fa-49cb-b68e-44ab467f5b02-maas-api-xks-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:30.736864 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.736863 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs8d4\" (UniqueName: \"kubernetes.io/projected/e4f8450f-01fa-49cb-b68e-44ab467f5b02-kube-api-access-hs8d4\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:30.959590 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.959518 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:53:30.960458 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.960436 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:53:30.988281 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:30.988243 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j"] Oct 30 12:53:31.235206 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:31.235102 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:31.369644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:31.369612 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" event={"ID":"e4f8450f-01fa-49cb-b68e-44ab467f5b02","Type":"ContainerDied","Data":"03430ccb4a478aeb4b63bbcb245660b13f0b5da2f4bf2d6d973b8a705a8af031"} Oct 30 12:53:31.369812 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:31.369670 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03430ccb4a478aeb4b63bbcb245660b13f0b5da2f4bf2d6d973b8a705a8af031" Oct 30 12:53:31.369812 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:31.369802 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz" Oct 30 12:53:31.447416 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:31.447389 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:31.668303 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:31.666255 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:31.814924 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:31.814892 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9"] Oct 30 12:53:32.061752 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:32.061661 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:32.267628 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:32.267593 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:32.766773 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:32.766737 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:32.998733 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:32.998698 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp"] Oct 30 12:53:32.998979 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:32.998940 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" podUID="1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" containerName="maas-api" containerID="cri-o://ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780" gracePeriod=30 Oct 30 12:53:33.175852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.175825 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:33.294617 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.294590 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:33.372585 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.372529 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-metrics-certs\") pod \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " Oct 30 12:53:33.372776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.372635 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-openshift-service-ca\") pod \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " Oct 30 12:53:33.372776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.372740 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-maas-api-tls\") pod \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " Oct 30 12:53:33.372893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.372780 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-kube-api-access-9c9sc\" (UniqueName: \"kubernetes.io/projected/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-kube-api-access-9c9sc\") pod \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\" (UID: \"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3\") " Oct 30 12:53:33.373095 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.373053 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-openshift-service-ca" pod "1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" (UID: "1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:33.373206 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.373183 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:33.375379 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.375354 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-metrics-certs" pod "1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" (UID: "1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:33.375471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.375382 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-kube-api-access-9c9sc" pod "1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" (UID: "1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3"). InnerVolumeSpecName "kube-api-access-9c9sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:33.375471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.375382 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-maas-api-tls" pod "1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" (UID: "1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:33.383728 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.383697 5515 generic.go:356] "Generic (PLEG): container finished" podID="1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" containerID="ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780" exitCode=0 Oct 30 12:53:33.383831 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.383742 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" Oct 30 12:53:33.383831 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.383773 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" event={"ID":"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3","Type":"ContainerDied","Data":"ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780"} Oct 30 12:53:33.383831 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.383827 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp" event={"ID":"1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3","Type":"ContainerDied","Data":"ed5c4ec09a8e1347324418266c4e50772dff5c42ed5364cf3ed0ce73fa6649a5"} Oct 30 12:53:33.383982 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.383851 5515 scope.go:122] "RemoveContainer" containerID="ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780" Oct 30 12:53:33.387887 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.387864 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:33.398106 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.398088 5515 scope.go:122] "RemoveContainer" containerID="ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780" Oct 30 12:53:33.398447 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:33.398426 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780\": container with ID starting with ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780 not found: ID does not exist" containerID="ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780" Oct 30 12:53:33.398497 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.398455 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780"} err="failed to get container status \"ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780\": rpc error: code = NotFound desc = could not find container \"ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780\": container with ID starting with ae991086b14f41bb65141b8e543668b9fb76caaeae7bae4b4fc24f23553cb780 not found: ID does not exist" Oct 30 12:53:33.440280 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.440253 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp"] Oct 30 12:53:33.443911 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.443892 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fb8b5ba5-5c64f644f8-z4whp"] Oct 30 12:53:33.474549 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.474521 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:33.474549 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.474544 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:33.474549 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.474553 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c9sc\" (UniqueName: \"kubernetes.io/projected/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3-kube-api-access-9c9sc\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:33.696825 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.696793 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:33.920813 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.920777 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:33.974698 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.974608 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz"] Oct 30 12:53:33.974882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:33.974825 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" containerID="cri-o://15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481" gracePeriod=30 Oct 30 12:53:34.001913 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.001888 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" path="/var/lib/kubelet/pods/1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3/volumes" Oct 30 12:53:34.129351 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.129315 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:34.669923 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.669881 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" podUID="ecc31286-ddda-485c-9581-28c31e0fd3b3" containerName="payload-pre-processing" containerID="cri-o://1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267" gracePeriod=25 Oct 30 12:53:34.935100 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.935034 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:34.990583 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.989646 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:34.993566 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.993538 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc31286-ddda-485c-9581-28c31e0fd3b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc31286-ddda-485c-9581-28c31e0fd3b3-plugins-config-volume\") pod \"ecc31286-ddda-485c-9581-28c31e0fd3b3\" (UID: \"ecc31286-ddda-485c-9581-28c31e0fd3b3\") " Oct 30 12:53:34.993709 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.993632 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc31286-ddda-485c-9581-28c31e0fd3b3-kube-api-access-ffdnx\" (UniqueName: \"kubernetes.io/projected/ecc31286-ddda-485c-9581-28c31e0fd3b3-kube-api-access-ffdnx\") pod \"ecc31286-ddda-485c-9581-28c31e0fd3b3\" (UID: \"ecc31286-ddda-485c-9581-28c31e0fd3b3\") " Oct 30 12:53:34.993929 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.993904 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc31286-ddda-485c-9581-28c31e0fd3b3-plugins-config-volume" pod "ecc31286-ddda-485c-9581-28c31e0fd3b3" (UID: "ecc31286-ddda-485c-9581-28c31e0fd3b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:34.994134 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.994116 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc31286-ddda-485c-9581-28c31e0fd3b3-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:34.997840 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:34.997812 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc31286-ddda-485c-9581-28c31e0fd3b3-kube-api-access-ffdnx" pod "ecc31286-ddda-485c-9581-28c31e0fd3b3" (UID: "ecc31286-ddda-485c-9581-28c31e0fd3b3"). InnerVolumeSpecName "kube-api-access-ffdnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:35.095318 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.095288 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffdnx\" (UniqueName: \"kubernetes.io/projected/ecc31286-ddda-485c-9581-28c31e0fd3b3-kube-api-access-ffdnx\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:35.192848 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.192769 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:35.399861 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.399823 5515 generic.go:356] "Generic (PLEG): container finished" podID="ecc31286-ddda-485c-9581-28c31e0fd3b3" containerID="1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267" exitCode=0 Oct 30 12:53:35.400014 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.399910 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" event={"ID":"ecc31286-ddda-485c-9581-28c31e0fd3b3","Type":"ContainerDied","Data":"1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267"} Oct 30 12:53:35.400014 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.399935 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" Oct 30 12:53:35.400014 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.399957 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6" event={"ID":"ecc31286-ddda-485c-9581-28c31e0fd3b3","Type":"ContainerDied","Data":"5c45fd312b8529743385f3b4aba89ae71449c86196f5812d6aa83b99dc052ddd"} Oct 30 12:53:35.400014 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.399981 5515 scope.go:122] "RemoveContainer" containerID="1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267" Oct 30 12:53:35.407118 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.407090 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:35.409047 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.409030 5515 scope.go:122] "RemoveContainer" containerID="1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267" Oct 30 12:53:35.409362 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:35.409343 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267\": container with ID starting with 1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267 not found: ID does not exist" containerID="1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267" Oct 30 12:53:35.409457 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.409370 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267"} err="failed to get container status \"1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267\": rpc error: code = NotFound desc = could not find container \"1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267\": container with ID starting with 1e4e62fe40e2e108e9481dd8b5571b998f1559c4eaa74b894814b0606264f267 not found: ID does not exist" Oct 30 12:53:35.454931 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.454867 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6"] Oct 30 12:53:35.467200 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.467173 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fb8b5ba5-b497879c5-9gqk6"] Oct 30 12:53:35.614316 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.614278 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz"] Oct 30 12:53:35.633459 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.633423 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:35.652406 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.652376 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt"] Oct 30 12:53:35.881352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.866619 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:35.988885 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:35.988845 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" podUID="4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1" containerName="payload-processing" containerID="cri-o://72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5" gracePeriod=25 Oct 30 12:53:36.000570 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.000541 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc31286-ddda-485c-9581-28c31e0fd3b3" path="/var/lib/kubelet/pods/ecc31286-ddda-485c-9581-28c31e0fd3b3/volumes" Oct 30 12:53:36.077774 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.077715 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:36.253037 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.253005 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:36.316971 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.316939 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-plugins-config-volume\") pod \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\" (UID: \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\") " Oct 30 12:53:36.317167 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.317003 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-kube-api-access-6vlnm\" (UniqueName: \"kubernetes.io/projected/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-kube-api-access-6vlnm\") pod \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\" (UID: \"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1\") " Oct 30 12:53:36.317374 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.317348 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-plugins-config-volume" pod "4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1" (UID: "4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:36.319258 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.319231 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-kube-api-access-6vlnm" pod "4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1" (UID: "4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1"). InnerVolumeSpecName "kube-api-access-6vlnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:36.406499 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.406464 5515 generic.go:356] "Generic (PLEG): container finished" podID="4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1" containerID="72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5" exitCode=0 Oct 30 12:53:36.406684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.406518 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" Oct 30 12:53:36.406684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.406552 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" event={"ID":"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1","Type":"ContainerDied","Data":"72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5"} Oct 30 12:53:36.406684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.406602 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j" event={"ID":"4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1","Type":"ContainerDied","Data":"0fe1ac57ac10651ab7a8fc998a2cdfc7767ca26a4321ba0fddc02a8d9e585535"} Oct 30 12:53:36.406684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.406635 5515 scope.go:122] "RemoveContainer" containerID="72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5" Oct 30 12:53:36.416670 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.416654 5515 scope.go:122] "RemoveContainer" containerID="72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5" Oct 30 12:53:36.416912 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:36.416896 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5\": container with ID starting with 72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5 not found: ID does not exist" containerID="72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5" Oct 30 12:53:36.416950 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.416919 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5"} err="failed to get container status \"72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5\": rpc error: code = NotFound desc = could not find container \"72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5\": container with ID starting with 72f96a971e69f1d118ac1436c46002865bd11c7aa5af0d3ebb78634306cb2ca5 not found: ID does not exist" Oct 30 12:53:36.418435 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.418418 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:36.418474 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.418438 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vlnm\" (UniqueName: \"kubernetes.io/projected/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1-kube-api-access-6vlnm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:36.489636 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.489597 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j"] Oct 30 12:53:36.496293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.496246 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f1c4045f-6f6bd678b8-smv6j"] Oct 30 12:53:36.679245 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.679172 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:36.817295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.816419 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" podUID="74990b01-7eaf-48bd-9a1f-12e3abb16556" containerName="payload-processing" containerID="cri-o://ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0" gracePeriod=25 Oct 30 12:53:36.844460 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.844430 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:53:36.889758 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:36.889727 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:37.072528 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.072502 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:37.126153 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.126123 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74990b01-7eaf-48bd-9a1f-12e3abb16556-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74990b01-7eaf-48bd-9a1f-12e3abb16556-plugins-config-volume\") pod \"74990b01-7eaf-48bd-9a1f-12e3abb16556\" (UID: \"74990b01-7eaf-48bd-9a1f-12e3abb16556\") " Oct 30 12:53:37.126350 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.126293 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74990b01-7eaf-48bd-9a1f-12e3abb16556-kube-api-access-js7dj\" (UniqueName: \"kubernetes.io/projected/74990b01-7eaf-48bd-9a1f-12e3abb16556-kube-api-access-js7dj\") pod \"74990b01-7eaf-48bd-9a1f-12e3abb16556\" (UID: \"74990b01-7eaf-48bd-9a1f-12e3abb16556\") " Oct 30 12:53:37.126519 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.126496 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74990b01-7eaf-48bd-9a1f-12e3abb16556-plugins-config-volume" pod "74990b01-7eaf-48bd-9a1f-12e3abb16556" (UID: "74990b01-7eaf-48bd-9a1f-12e3abb16556"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:37.126684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.126664 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74990b01-7eaf-48bd-9a1f-12e3abb16556-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:37.128408 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.128377 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74990b01-7eaf-48bd-9a1f-12e3abb16556-kube-api-access-js7dj" pod "74990b01-7eaf-48bd-9a1f-12e3abb16556" (UID: "74990b01-7eaf-48bd-9a1f-12e3abb16556"). InnerVolumeSpecName "kube-api-access-js7dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:37.228160 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.228081 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7dj\" (UniqueName: \"kubernetes.io/projected/74990b01-7eaf-48bd-9a1f-12e3abb16556-kube-api-access-js7dj\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:37.391526 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.391491 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s"] Oct 30 12:53:37.391757 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.391733 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" podUID="6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" containerName="maas-api" containerID="cri-o://77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a" gracePeriod=30 Oct 30 12:53:37.417169 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.417139 5515 generic.go:356] "Generic (PLEG): container finished" podID="74990b01-7eaf-48bd-9a1f-12e3abb16556" containerID="ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0" exitCode=0 Oct 30 12:53:37.417447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.417193 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" Oct 30 12:53:37.417447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.417218 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" event={"ID":"74990b01-7eaf-48bd-9a1f-12e3abb16556","Type":"ContainerDied","Data":"ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0"} Oct 30 12:53:37.417447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.417323 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9" event={"ID":"74990b01-7eaf-48bd-9a1f-12e3abb16556","Type":"ContainerDied","Data":"ece2f246a11a61635674ddf8c8bf7571f786715a686642ab18548d4fffabdd59"} Oct 30 12:53:37.417447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.417350 5515 scope.go:122] "RemoveContainer" containerID="ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0" Oct 30 12:53:37.490390 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.490313 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9"] Oct 30 12:53:37.495246 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.495221 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-873e4f54-84b4fdbd5d-wjtr9"] Oct 30 12:53:37.504735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.504715 5515 scope.go:122] "RemoveContainer" containerID="ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0" Oct 30 12:53:37.505059 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:37.505039 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0\": container with ID starting with ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0 not found: ID does not exist" containerID="ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0" Oct 30 12:53:37.505130 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.505066 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0"} err="failed to get container status \"ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0\": rpc error: code = NotFound desc = could not find container \"ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0\": container with ID starting with ee63f7a266567d513ddab60b19503f4031c96044a051bb5af3724aff491e6cf0 not found: ID does not exist" Oct 30 12:53:37.559710 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.559681 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:37.685391 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.685359 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:37.740144 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.739196 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-maas-api-tls\") pod \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " Oct 30 12:53:37.740144 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.739387 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-openshift-service-ca\") pod \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " Oct 30 12:53:37.740144 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.739446 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-metrics-certs\") pod \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " Oct 30 12:53:37.740144 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.739479 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-kube-api-access-f48bn\" (UniqueName: \"kubernetes.io/projected/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-kube-api-access-f48bn\") pod \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\" (UID: \"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e\") " Oct 30 12:53:37.740144 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.739846 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-openshift-service-ca" pod "6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" (UID: "6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:37.741612 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.741561 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-maas-api-tls" pod "6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" (UID: "6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:37.742051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.742028 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-metrics-certs" pod "6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" (UID: "6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:53:37.742116 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.742098 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-kube-api-access-f48bn" pod "6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" (UID: "6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e"). InnerVolumeSpecName "kube-api-access-f48bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:37.771851 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.771812 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:37.840445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.840413 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:37.840445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.840440 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:37.840445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.840450 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f48bn\" (UniqueName: \"kubernetes.io/projected/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-kube-api-access-f48bn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:37.840684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.840459 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:37.922413 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922378 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq"] Oct 30 12:53:37.922837 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922824 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" containerName="maas-api" Oct 30 12:53:37.922888 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922839 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" containerName="maas-api" Oct 30 12:53:37.922888 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922849 5515 cpu_manager.go:414] "containerMap: removing container" podUID="74990b01-7eaf-48bd-9a1f-12e3abb16556" containerName="payload-processing" Oct 30 12:53:37.922888 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922856 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74990b01-7eaf-48bd-9a1f-12e3abb16556" containerName="payload-processing" Oct 30 12:53:37.922888 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922868 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ecc31286-ddda-485c-9581-28c31e0fd3b3" containerName="payload-pre-processing" Oct 30 12:53:37.922888 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922873 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc31286-ddda-485c-9581-28c31e0fd3b3" containerName="payload-pre-processing" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922890 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1" containerName="payload-processing" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922895 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1" containerName="payload-processing" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922906 5515 cpu_manager.go:414] "containerMap: removing container" podUID="e4f8450f-01fa-49cb-b68e-44ab467f5b02" containerName="revoke-keys" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922915 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f8450f-01fa-49cb-b68e-44ab467f5b02" containerName="revoke-keys" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922925 5515 cpu_manager.go:414] "containerMap: removing container" podUID="6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" containerName="maas-api" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.922930 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" containerName="maas-api" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.923018 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7f0f9e-ec9b-4d96-a066-dab0a174a3e3" containerName="maas-api" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.923026 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="74990b01-7eaf-48bd-9a1f-12e3abb16556" containerName="payload-processing" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.923035 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc31286-ddda-485c-9581-28c31e0fd3b3" containerName="payload-pre-processing" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.923046 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1" containerName="payload-processing" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.923054 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f8450f-01fa-49cb-b68e-44ab467f5b02" containerName="revoke-keys" Oct 30 12:53:37.923138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.923063 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" containerName="maas-api" Oct 30 12:53:37.927101 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.927079 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.941772 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.941748 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.941884 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.941852 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.941956 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.941889 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.942039 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.942014 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwnk\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-kube-api-access-2jwnk\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.942100 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.942087 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.942228 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.942208 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.942295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.942255 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.942376 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.942360 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f163fa-56a6-411a-802b-d421d7b68e6e-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.942473 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.942454 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:37.942775 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.942752 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7a8b03-openshift-default-dockercfg-mbrk9\"" Oct 30 12:53:37.957351 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:37.957322 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq"] Oct 30 12:53:38.008478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.008390 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1" path="/var/lib/kubelet/pods/4d31c1bf-de1b-4c72-8560-0dfe4a7bedf1/volumes" Oct 30 12:53:38.008869 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.008849 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74990b01-7eaf-48bd-9a1f-12e3abb16556" path="/var/lib/kubelet/pods/74990b01-7eaf-48bd-9a1f-12e3abb16556/volumes" Oct 30 12:53:38.043117 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043085 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f163fa-56a6-411a-802b-d421d7b68e6e-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043129 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043148 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043165 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043194 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043240 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwnk\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-kube-api-access-2jwnk\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043313 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043592 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043375 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043592 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043408 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043722 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043696 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043814 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043771 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.043917 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.043895 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.044029 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.044006 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.044123 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.044101 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f163fa-56a6-411a-802b-d421d7b68e6e-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.045633 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.045612 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.046070 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.046050 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.066171 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.066141 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.066353 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.066332 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwnk\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-kube-api-access-2jwnk\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.099691 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.099659 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:38.150128 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.150098 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz"] Oct 30 12:53:38.152968 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.152943 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc59b5af-blfxz"] Oct 30 12:53:38.249727 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.249695 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:38.305107 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.304979 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t"] Oct 30 12:53:38.305356 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.305190 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" containerID="cri-o://31f5a875c9b4aef3d216d583d392ceda891657a0e973b6c370f8e03d8b6cdaac" gracePeriod=30 Oct 30 12:53:38.312002 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.311823 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:38.440625 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.440594 5515 generic.go:356] "Generic (PLEG): container finished" podID="6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" containerID="77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a" exitCode=0 Oct 30 12:53:38.440810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.440710 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" event={"ID":"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e","Type":"ContainerDied","Data":"77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a"} Oct 30 12:53:38.440810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.440747 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" event={"ID":"6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e","Type":"ContainerDied","Data":"f71b94a1384cb5b13ec0a20fb34b25b9f8ba9b47e5a0062b199428c584d95fa5"} Oct 30 12:53:38.440810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.440776 5515 scope.go:122] "RemoveContainer" containerID="77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a" Oct 30 12:53:38.440999 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.440904 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s" Oct 30 12:53:38.454145 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.454127 5515 scope.go:122] "RemoveContainer" containerID="77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a" Oct 30 12:53:38.454483 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:38.454459 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a\": container with ID starting with 77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a not found: ID does not exist" containerID="77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a" Oct 30 12:53:38.454601 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.454491 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a"} err="failed to get container status \"77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a\": rpc error: code = NotFound desc = could not find container \"77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a\": container with ID starting with 77bb48d63fb0c9008ce09b722f310ec454b05d07c9fd0396497b71fa7285538a not found: ID does not exist" Oct 30 12:53:38.500146 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.500114 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s"] Oct 30 12:53:38.502101 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.502070 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc59b5af-6b5df95c74-kqz4s"] Oct 30 12:53:38.526005 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.525932 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:38.583427 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.583405 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq"] Oct 30 12:53:38.586148 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:38.586062 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f163fa_56a6_411a_802b_d421d7b68e6e.slice/crio-843c845ac2fba0488644d63c4f5a397af986c7ced12a737112bbe106fa5755a0 WatchSource:0}: Error finding container 843c845ac2fba0488644d63c4f5a397af986c7ced12a737112bbe106fa5755a0: Status 404 returned error can't find the container with id 843c845ac2fba0488644d63c4f5a397af986c7ced12a737112bbe106fa5755a0 Oct 30 12:53:38.588761 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.588732 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:38.588850 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.588795 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:38.588850 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.588825 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:38.640906 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.640877 5515 patch_prober.go:29] interesting pod/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 12:53:38.641044 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:38.640932 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:53:39.232206 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.232176 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:53:39.257036 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.256969 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-envoy\") pod \"50b3d16b-57f6-4a07-9903-2a002a10761b\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " Oct 30 12:53:39.257036 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257011 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-token\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-token\") pod \"50b3d16b-57f6-4a07-9903-2a002a10761b\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " Oct 30 12:53:39.257247 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257049 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-socket\") pod \"50b3d16b-57f6-4a07-9903-2a002a10761b\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " Oct 30 12:53:39.257247 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257170 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-certs\") pod \"50b3d16b-57f6-4a07-9903-2a002a10761b\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " Oct 30 12:53:39.257386 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257292 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-data\") pod \"50b3d16b-57f6-4a07-9903-2a002a10761b\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " Oct 30 12:53:39.257386 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257351 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-socket" pod "50b3d16b-57f6-4a07-9903-2a002a10761b" (UID: "50b3d16b-57f6-4a07-9903-2a002a10761b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:39.257490 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257386 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b3d16b-57f6-4a07-9903-2a002a10761b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b3d16b-57f6-4a07-9903-2a002a10761b-istiod-ca-cert\") pod \"50b3d16b-57f6-4a07-9903-2a002a10761b\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " Oct 30 12:53:39.257490 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257435 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-podinfo\") pod \"50b3d16b-57f6-4a07-9903-2a002a10761b\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " Oct 30 12:53:39.257490 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257462 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-certs" pod "50b3d16b-57f6-4a07-9903-2a002a10761b" (UID: "50b3d16b-57f6-4a07-9903-2a002a10761b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:39.257640 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257490 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-data" pod "50b3d16b-57f6-4a07-9903-2a002a10761b" (UID: "50b3d16b-57f6-4a07-9903-2a002a10761b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:39.257640 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257500 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-kube-api-access-f8g5n\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-kube-api-access-f8g5n\") pod \"50b3d16b-57f6-4a07-9903-2a002a10761b\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " Oct 30 12:53:39.257640 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257621 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-credential-socket\") pod \"50b3d16b-57f6-4a07-9903-2a002a10761b\" (UID: \"50b3d16b-57f6-4a07-9903-2a002a10761b\") " Oct 30 12:53:39.257785 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257704 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b3d16b-57f6-4a07-9903-2a002a10761b-istiod-ca-cert" pod "50b3d16b-57f6-4a07-9903-2a002a10761b" (UID: "50b3d16b-57f6-4a07-9903-2a002a10761b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:39.257887 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.257868 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-credential-socket" pod "50b3d16b-57f6-4a07-9903-2a002a10761b" (UID: "50b3d16b-57f6-4a07-9903-2a002a10761b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:39.258040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.258024 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:39.258116 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.258041 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:39.258116 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.258051 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:39.258116 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.258059 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:39.258116 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.258067 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b3d16b-57f6-4a07-9903-2a002a10761b-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:39.259514 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.259490 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-podinfo" pod "50b3d16b-57f6-4a07-9903-2a002a10761b" (UID: "50b3d16b-57f6-4a07-9903-2a002a10761b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:53:39.259684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.259661 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-token" pod "50b3d16b-57f6-4a07-9903-2a002a10761b" (UID: "50b3d16b-57f6-4a07-9903-2a002a10761b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:39.259684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.259673 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-kube-api-access-f8g5n" pod "50b3d16b-57f6-4a07-9903-2a002a10761b" (UID: "50b3d16b-57f6-4a07-9903-2a002a10761b"). InnerVolumeSpecName "kube-api-access-f8g5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:39.259797 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.259663 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-envoy" pod "50b3d16b-57f6-4a07-9903-2a002a10761b" (UID: "50b3d16b-57f6-4a07-9903-2a002a10761b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:39.308369 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.308332 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:39.358796 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.358763 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:39.358796 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.358797 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8g5n\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-kube-api-access-f8g5n\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:39.359019 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.358812 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:39.359019 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.358825 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b3d16b-57f6-4a07-9903-2a002a10761b-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:39.448825 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.448795 5515 generic.go:356] "Generic (PLEG): container finished" podID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerID="15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481" exitCode=0 Oct 30 12:53:39.449017 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.448850 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" Oct 30 12:53:39.449017 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.448881 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" event={"ID":"50b3d16b-57f6-4a07-9903-2a002a10761b","Type":"ContainerDied","Data":"15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481"} Oct 30 12:53:39.449017 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.448934 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz" event={"ID":"50b3d16b-57f6-4a07-9903-2a002a10761b","Type":"ContainerDied","Data":"788a5b8c0c433d2853894bc562ba2ea22a5b6b7c45bd4d2ab5700e54fa021d21"} Oct 30 12:53:39.449017 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.448965 5515 scope.go:122] "RemoveContainer" containerID="15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481" Oct 30 12:53:39.450412 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.450389 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" event={"ID":"52f163fa-56a6-411a-802b-d421d7b68e6e","Type":"ContainerStarted","Data":"983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c"} Oct 30 12:53:39.450478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.450437 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" event={"ID":"52f163fa-56a6-411a-802b-d421d7b68e6e","Type":"ContainerStarted","Data":"843c845ac2fba0488644d63c4f5a397af986c7ced12a737112bbe106fa5755a0"} Oct 30 12:53:39.458595 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.458574 5515 scope.go:122] "RemoveContainer" containerID="f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a" Oct 30 12:53:39.466760 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.466741 5515 scope.go:122] "RemoveContainer" containerID="15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481" Oct 30 12:53:39.467015 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:39.466998 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481\": container with ID starting with 15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481 not found: ID does not exist" containerID="15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481" Oct 30 12:53:39.467055 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.467024 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481"} err="failed to get container status \"15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481\": rpc error: code = NotFound desc = could not find container \"15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481\": container with ID starting with 15840a2bf6aa664f5a934aebd90c2b33ad1fe092725a13e1ac6496c5c859e481 not found: ID does not exist" Oct 30 12:53:39.467055 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.467039 5515 scope.go:122] "RemoveContainer" containerID="f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a" Oct 30 12:53:39.467255 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:39.467239 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a\": container with ID starting with f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a not found: ID does not exist" containerID="f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a" Oct 30 12:53:39.467387 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.467261 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a"} err="failed to get container status \"f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a\": rpc error: code = NotFound desc = could not find container \"f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a\": container with ID starting with f97afa6d2f10ac2137e18c39879923fff9666fa4f13df3ffa8de27a89fe3523a not found: ID does not exist" Oct 30 12:53:39.515252 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.515173 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz"] Oct 30 12:53:39.525669 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.525641 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fb8b5ba5-gw-openshift-default-646b74cfb5-4rvwz"] Oct 30 12:53:39.531650 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.531624 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:39.573535 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.573485 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" podStartSLOduration=2.573473511 podStartE2EDuration="2.573473511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:39.57272665 +0000 UTC m=+1237.699527548" watchObservedRunningTime="2026-10-30 12:53:39.573473511 +0000 UTC m=+1237.700274424" Oct 30 12:53:39.694759 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.694724 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh"] Oct 30 12:53:39.695454 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.695431 5515 cpu_manager.go:414] "containerMap: removing container" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" Oct 30 12:53:39.695454 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.695454 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" Oct 30 12:53:39.695611 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.695591 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" Oct 30 12:53:39.695647 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.695610 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" Oct 30 12:53:39.700933 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.700911 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.721884 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.721210 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb86fd-openshift-default-dockercfg-bqgfs\"" Oct 30 12:53:39.723988 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.723963 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh"] Oct 30 12:53:39.733425 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.733399 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:39.762959 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.762928 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.763147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.763002 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.763147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.763049 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.763246 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.763143 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d51232-79de-44ac-a895-4553f87ed4e8-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.763246 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.763207 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.763347 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.763328 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.763382 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.763362 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2d4\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-kube-api-access-zr2d4\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.763443 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.763422 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.763501 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.763463 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.864597 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.864508 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.864597 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.864554 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d51232-79de-44ac-a895-4553f87ed4e8-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.864839 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.864608 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.864839 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.864631 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.864839 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.864772 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2d4\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-kube-api-access-zr2d4\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.865013 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.864840 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.865013 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.864909 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.865013 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.864912 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.865013 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.864948 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.865222 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.865027 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.865222 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.865130 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.865349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.865234 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.865349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.865262 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.865439 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.865371 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d51232-79de-44ac-a895-4553f87ed4e8-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.867225 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.867201 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.867459 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.867442 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.898596 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.898570 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:39.898745 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:39.898706 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2d4\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-kube-api-access-zr2d4\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:40.003011 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.001311 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:40.008659 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.008628 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" path="/var/lib/kubelet/pods/50b3d16b-57f6-4a07-9903-2a002a10761b/volumes" Oct 30 12:53:40.012739 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.012709 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e" path="/var/lib/kubelet/pods/6630deb8-bbdc-4b8c-88e0-da0fd2f6c31e/volumes" Oct 30 12:53:40.013644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.013620 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f8450f-01fa-49cb-b68e-44ab467f5b02" path="/var/lib/kubelet/pods/e4f8450f-01fa-49cb-b68e-44ab467f5b02/volumes" Oct 30 12:53:40.014314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.014290 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:40.203617 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.203581 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:40.250077 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.250047 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:40.255303 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.255261 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:40.347835 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.347810 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh"] Oct 30 12:53:40.349918 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:40.349888 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d51232_79de_44ac_a895_4553f87ed4e8.slice/crio-04defa35d964df2212450cead8978620a75fe091aec47df712285a26883386b0 WatchSource:0}: Error finding container 04defa35d964df2212450cead8978620a75fe091aec47df712285a26883386b0: Status 404 returned error can't find the container with id 04defa35d964df2212450cead8978620a75fe091aec47df712285a26883386b0 Oct 30 12:53:40.352663 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.352621 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:40.352787 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.352711 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:40.352787 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.352757 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:53:40.433814 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.433786 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:53:40.468685 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.468583 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" event={"ID":"a7d51232-79de-44ac-a895-4553f87ed4e8","Type":"ContainerStarted","Data":"442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d"} Oct 30 12:53:40.468685 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.468678 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" event={"ID":"a7d51232-79de-44ac-a895-4553f87ed4e8","Type":"ContainerStarted","Data":"04defa35d964df2212450cead8978620a75fe091aec47df712285a26883386b0"} Oct 30 12:53:40.470390 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.470364 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:40.471428 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.471308 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:53:40.514626 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.514567 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" podStartSLOduration=1.5145546680000002 podStartE2EDuration="1.514554668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:40.513363077 +0000 UTC m=+1238.640163976" watchObservedRunningTime="2026-10-30 12:53:40.514554668 +0000 UTC m=+1238.641355565" Oct 30 12:53:40.614934 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.614892 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" podUID="0d440fd2-f9f6-479f-bb17-471ea645e4a8" containerName="payload-processing" containerID="cri-o://b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7" gracePeriod=25 Oct 30 12:53:40.652662 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.652594 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" podUID="03676fec-d216-480d-917e-6d084b220599" containerName="payload-pre-processing" containerID="cri-o://4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb" gracePeriod=25 Oct 30 12:53:40.895880 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.895853 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:40.921592 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.921561 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:40.990935 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.990896 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d440fd2-f9f6-479f-bb17-471ea645e4a8-kube-api-access-h9cqq\" (UniqueName: \"kubernetes.io/projected/0d440fd2-f9f6-479f-bb17-471ea645e4a8-kube-api-access-h9cqq\") pod \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\" (UID: \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\") " Oct 30 12:53:40.991098 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.990997 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03676fec-d216-480d-917e-6d084b220599-kube-api-access-swvdg\" (UniqueName: \"kubernetes.io/projected/03676fec-d216-480d-917e-6d084b220599-kube-api-access-swvdg\") pod \"03676fec-d216-480d-917e-6d084b220599\" (UID: \"03676fec-d216-480d-917e-6d084b220599\") " Oct 30 12:53:40.991147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.991103 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d440fd2-f9f6-479f-bb17-471ea645e4a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d440fd2-f9f6-479f-bb17-471ea645e4a8-plugins-config-volume\") pod \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\" (UID: \"0d440fd2-f9f6-479f-bb17-471ea645e4a8\") " Oct 30 12:53:40.991198 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.991142 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03676fec-d216-480d-917e-6d084b220599-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03676fec-d216-480d-917e-6d084b220599-plugins-config-volume\") pod \"03676fec-d216-480d-917e-6d084b220599\" (UID: \"03676fec-d216-480d-917e-6d084b220599\") " Oct 30 12:53:40.991612 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.991581 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d440fd2-f9f6-479f-bb17-471ea645e4a8-plugins-config-volume" pod "0d440fd2-f9f6-479f-bb17-471ea645e4a8" (UID: "0d440fd2-f9f6-479f-bb17-471ea645e4a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:40.991743 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.991580 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03676fec-d216-480d-917e-6d084b220599-plugins-config-volume" pod "03676fec-d216-480d-917e-6d084b220599" (UID: "03676fec-d216-480d-917e-6d084b220599"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:40.993672 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.993646 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d440fd2-f9f6-479f-bb17-471ea645e4a8-kube-api-access-h9cqq" pod "0d440fd2-f9f6-479f-bb17-471ea645e4a8" (UID: "0d440fd2-f9f6-479f-bb17-471ea645e4a8"). InnerVolumeSpecName "kube-api-access-h9cqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:40.993763 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:40.993732 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03676fec-d216-480d-917e-6d084b220599-kube-api-access-swvdg" pod "03676fec-d216-480d-917e-6d084b220599" (UID: "03676fec-d216-480d-917e-6d084b220599"). InnerVolumeSpecName "kube-api-access-swvdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:41.014829 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.014758 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:41.019787 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.019766 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:41.061170 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.061127 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:41.092935 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.092907 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03676fec-d216-480d-917e-6d084b220599-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:41.092935 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.092932 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9cqq\" (UniqueName: \"kubernetes.io/projected/0d440fd2-f9f6-479f-bb17-471ea645e4a8-kube-api-access-h9cqq\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:41.092935 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.092942 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvdg\" (UniqueName: \"kubernetes.io/projected/03676fec-d216-480d-917e-6d084b220599-kube-api-access-swvdg\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:41.093155 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.092952 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d440fd2-f9f6-479f-bb17-471ea645e4a8-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:41.271934 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.271841 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:41.428575 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.423947 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5"] Oct 30 12:53:41.434828 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.434785 5515 cpu_manager.go:414] "containerMap: removing container" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" Oct 30 12:53:41.434828 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.434815 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b3d16b-57f6-4a07-9903-2a002a10761b" containerName="istio-proxy" Oct 30 12:53:41.435064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.434860 5515 cpu_manager.go:414] "containerMap: removing container" podUID="03676fec-d216-480d-917e-6d084b220599" containerName="payload-pre-processing" Oct 30 12:53:41.435064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.434868 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03676fec-d216-480d-917e-6d084b220599" containerName="payload-pre-processing" Oct 30 12:53:41.435064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.434895 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0d440fd2-f9f6-479f-bb17-471ea645e4a8" containerName="payload-processing" Oct 30 12:53:41.435064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.434902 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d440fd2-f9f6-479f-bb17-471ea645e4a8" containerName="payload-processing" Oct 30 12:53:41.435064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.435002 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="03676fec-d216-480d-917e-6d084b220599" containerName="payload-pre-processing" Oct 30 12:53:41.435064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.435016 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d440fd2-f9f6-479f-bb17-471ea645e4a8" containerName="payload-processing" Oct 30 12:53:41.439475 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.439450 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:41.478289 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.465130 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-551dd1-dockercfg-6rsbl\"" Oct 30 12:53:41.478289 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.465399 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-551dd1\"" Oct 30 12:53:41.478289 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.468785 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5"] Oct 30 12:53:41.490980 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.490941 5515 generic.go:356] "Generic (PLEG): container finished" podID="0d440fd2-f9f6-479f-bb17-471ea645e4a8" containerID="b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7" exitCode=0 Oct 30 12:53:41.491135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.491001 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" Oct 30 12:53:41.491135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.491025 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" event={"ID":"0d440fd2-f9f6-479f-bb17-471ea645e4a8","Type":"ContainerDied","Data":"b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7"} Oct 30 12:53:41.491135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.491087 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz" event={"ID":"0d440fd2-f9f6-479f-bb17-471ea645e4a8","Type":"ContainerDied","Data":"314cdab3cc0a77a035b0824ae305dab0fdb0488fc6cb67e45e9d7a00fcc0bac5"} Oct 30 12:53:41.491135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.491126 5515 scope.go:122] "RemoveContainer" containerID="b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7" Oct 30 12:53:41.492857 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.492833 5515 generic.go:356] "Generic (PLEG): container finished" podID="03676fec-d216-480d-917e-6d084b220599" containerID="4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb" exitCode=0 Oct 30 12:53:41.492950 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.492904 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" event={"ID":"03676fec-d216-480d-917e-6d084b220599","Type":"ContainerDied","Data":"4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb"} Oct 30 12:53:41.493013 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.492960 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" event={"ID":"03676fec-d216-480d-917e-6d084b220599","Type":"ContainerDied","Data":"8bd7871c9e1d30d99e30c10a1dc8023faceba8b4c316820f84c0b5c4280fc47f"} Oct 30 12:53:41.493611 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.493437 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:41.493863 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.493749 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt" Oct 30 12:53:41.495644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.494531 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:53:41.499337 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.499313 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2cr\" (UniqueName: \"kubernetes.io/projected/58a3c214-d603-45e7-8011-76b3e5d8bc67-kube-api-access-nd2cr\") pod \"payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5\" (UID: \"58a3c214-d603-45e7-8011-76b3e5d8bc67\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:41.499480 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.499460 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a3c214-d603-45e7-8011-76b3e5d8bc67-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5\" (UID: \"58a3c214-d603-45e7-8011-76b3e5d8bc67\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:41.502986 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.502969 5515 scope.go:122] "RemoveContainer" containerID="b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7" Oct 30 12:53:41.503705 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:41.503665 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7\": container with ID starting with b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7 not found: ID does not exist" containerID="b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7" Oct 30 12:53:41.503792 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.503717 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7"} err="failed to get container status \"b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7\": rpc error: code = NotFound desc = could not find container \"b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7\": container with ID starting with b077ef9d261474291109df19ad26c71ed5710d8b9601c3b3a6210e7550a15fa7 not found: ID does not exist" Oct 30 12:53:41.503792 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.503739 5515 scope.go:122] "RemoveContainer" containerID="4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb" Oct 30 12:53:41.512338 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.512316 5515 scope.go:122] "RemoveContainer" containerID="4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb" Oct 30 12:53:41.512622 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:41.512599 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb\": container with ID starting with 4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb not found: ID does not exist" containerID="4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb" Oct 30 12:53:41.512693 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.512627 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb"} err="failed to get container status \"4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb\": rpc error: code = NotFound desc = could not find container \"4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb\": container with ID starting with 4415616c24197b73a44dfc9a8ed77d13fced5fd4a6aeb1ce0b708f6453afd1eb not found: ID does not exist" Oct 30 12:53:41.559178 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.559097 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:41.600222 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.600166 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2cr\" (UniqueName: \"kubernetes.io/projected/58a3c214-d603-45e7-8011-76b3e5d8bc67-kube-api-access-nd2cr\") pod \"payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5\" (UID: \"58a3c214-d603-45e7-8011-76b3e5d8bc67\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:41.600461 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.600385 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a3c214-d603-45e7-8011-76b3e5d8bc67-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5\" (UID: \"58a3c214-d603-45e7-8011-76b3e5d8bc67\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:41.601053 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.601033 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a3c214-d603-45e7-8011-76b3e5d8bc67-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5\" (UID: \"58a3c214-d603-45e7-8011-76b3e5d8bc67\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:41.640316 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.640259 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2cr\" (UniqueName: \"kubernetes.io/projected/58a3c214-d603-45e7-8011-76b3e5d8bc67-kube-api-access-nd2cr\") pod \"payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5\" (UID: \"58a3c214-d603-45e7-8011-76b3e5d8bc67\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:41.662947 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.662911 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt"] Oct 30 12:53:41.666810 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.666784 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cc59b5af-75b6d87c8c-kjpjt"] Oct 30 12:53:41.703225 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.703192 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz"] Oct 30 12:53:41.712785 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.712748 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc59b5af-c8cb45d5d-cpdjz"] Oct 30 12:53:41.755715 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.755681 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:41.813222 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.811959 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd"] Oct 30 12:53:41.817292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.817250 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:41.840076 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.839261 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-551dd1-serving-cert\"" Oct 30 12:53:41.840076 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.839411 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dlqhq\"" Oct 30 12:53:41.840076 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.840031 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 12:53:41.841200 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.841181 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd"] Oct 30 12:53:41.903983 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.903949 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-metrics-certs\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:41.904140 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.904060 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w49b\" (UniqueName: \"kubernetes.io/projected/9870674e-3d75-4501-a5ce-4040294edd31-kube-api-access-6w49b\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:41.904140 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.904130 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-maas-api-tls\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:41.904205 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.904168 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9870674e-3d75-4501-a5ce-4040294edd31-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:41.982032 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:41.982001 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:42.000664 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.000633 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03676fec-d216-480d-917e-6d084b220599" path="/var/lib/kubelet/pods/03676fec-d216-480d-917e-6d084b220599/volumes" Oct 30 12:53:42.001021 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.001009 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d440fd2-f9f6-479f-bb17-471ea645e4a8" path="/var/lib/kubelet/pods/0d440fd2-f9f6-479f-bb17-471ea645e4a8/volumes" Oct 30 12:53:42.004904 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.004882 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-maas-api-tls\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:42.005034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.004924 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9870674e-3d75-4501-a5ce-4040294edd31-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:42.005034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.004984 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-metrics-certs\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:42.005034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.005027 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w49b\" (UniqueName: \"kubernetes.io/projected/9870674e-3d75-4501-a5ce-4040294edd31-kube-api-access-6w49b\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:42.005669 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.005649 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9870674e-3d75-4501-a5ce-4040294edd31-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:42.007444 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.007422 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-metrics-certs\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:42.007525 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.007484 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-maas-api-tls\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:42.027864 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.027838 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w49b\" (UniqueName: \"kubernetes.io/projected/9870674e-3d75-4501-a5ce-4040294edd31-kube-api-access-6w49b\") pod \"maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:42.065148 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:42.065072 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a3c214_d603_45e7_8011_76b3e5d8bc67.slice/crio-ba1395513269b92b194a07a5367d5c807aaf0969c73c4a5810002345d659a535 WatchSource:0}: Error finding container ba1395513269b92b194a07a5367d5c807aaf0969c73c4a5810002345d659a535: Status 404 returned error can't find the container with id ba1395513269b92b194a07a5367d5c807aaf0969c73c4a5810002345d659a535 Oct 30 12:53:42.066877 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.066570 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5"] Oct 30 12:53:42.146356 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.146326 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:42.183558 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.183520 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:42.453521 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.453500 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd"] Oct 30 12:53:42.454748 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:42.454721 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9870674e_3d75_4501_a5ce_4040294edd31.slice/crio-ba79f542136bd9739e218999c41aeb4357dafc8c0ecc4bced7a4f503b35c5863 WatchSource:0}: Error finding container ba79f542136bd9739e218999c41aeb4357dafc8c0ecc4bced7a4f503b35c5863: Status 404 returned error can't find the container with id ba79f542136bd9739e218999c41aeb4357dafc8c0ecc4bced7a4f503b35c5863 Oct 30 12:53:42.499432 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.499397 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" event={"ID":"9870674e-3d75-4501-a5ce-4040294edd31","Type":"ContainerStarted","Data":"ba79f542136bd9739e218999c41aeb4357dafc8c0ecc4bced7a4f503b35c5863"} Oct 30 12:53:42.501638 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.501613 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" event={"ID":"58a3c214-d603-45e7-8011-76b3e5d8bc67","Type":"ContainerStarted","Data":"8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61"} Oct 30 12:53:42.509083 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.501827 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" event={"ID":"58a3c214-d603-45e7-8011-76b3e5d8bc67","Type":"ContainerStarted","Data":"ba1395513269b92b194a07a5367d5c807aaf0969c73c4a5810002345d659a535"} Oct 30 12:53:42.509083 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.502171 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:42.544003 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.543955 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" podStartSLOduration=1.5439417899999999 podStartE2EDuration="1.54394179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:42.54304659 +0000 UTC m=+1240.669847491" watchObservedRunningTime="2026-10-30 12:53:42.54394179 +0000 UTC m=+1240.670742719" Oct 30 12:53:42.873377 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:42.873331 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:43.080374 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.080288 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:43.314551 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.314511 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:43.519094 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.519064 5515 generic.go:356] "Generic (PLEG): container finished" podID="d5d8350e-fb75-4613-ba55-717b478af302" containerID="31f5a875c9b4aef3d216d583d392ceda891657a0e973b6c370f8e03d8b6cdaac" exitCode=0 Oct 30 12:53:43.519543 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.519141 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" event={"ID":"d5d8350e-fb75-4613-ba55-717b478af302","Type":"ContainerDied","Data":"31f5a875c9b4aef3d216d583d392ceda891657a0e973b6c370f8e03d8b6cdaac"} Oct 30 12:53:43.519543 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.519219 5515 scope.go:122] "RemoveContainer" containerID="afa5a7a4d9161b0fdaba18827684dff25d874cae22490d89c52745e145e94d36" Oct 30 12:53:43.521039 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.521014 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" event={"ID":"9870674e-3d75-4501-a5ce-4040294edd31","Type":"ContainerStarted","Data":"98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522"} Oct 30 12:53:43.575462 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.575436 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:53:43.601580 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.601495 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" podStartSLOduration=2.289662338 podStartE2EDuration="2.601480987s" totalImagesPullingTime="311.818649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:43.599648735 +0000 UTC m=+1241.726449632" watchObservedRunningTime="2026-10-30 12:53:43.601480987 +0000 UTC m=+1241.728281884" Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.634967 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.640650 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-socket\") pod \"d5d8350e-fb75-4613-ba55-717b478af302\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.640725 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-envoy\") pod \"d5d8350e-fb75-4613-ba55-717b478af302\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.640783 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5d8350e-fb75-4613-ba55-717b478af302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8350e-fb75-4613-ba55-717b478af302-istio-podinfo\") pod \"d5d8350e-fb75-4613-ba55-717b478af302\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.640864 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-istio-token\") pod \"d5d8350e-fb75-4613-ba55-717b478af302\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.640908 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-kube-api-access-4mml6\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-kube-api-access-4mml6\") pod \"d5d8350e-fb75-4613-ba55-717b478af302\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.640913 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-socket" pod "d5d8350e-fb75-4613-ba55-717b478af302" (UID: "d5d8350e-fb75-4613-ba55-717b478af302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.640939 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d8350e-fb75-4613-ba55-717b478af302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8350e-fb75-4613-ba55-717b478af302-istiod-ca-cert\") pod \"d5d8350e-fb75-4613-ba55-717b478af302\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.641736 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-credential-socket\") pod \"d5d8350e-fb75-4613-ba55-717b478af302\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.641834 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-certs\") pod \"d5d8350e-fb75-4613-ba55-717b478af302\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.641886 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-data\") pod \"d5d8350e-fb75-4613-ba55-717b478af302\" (UID: \"d5d8350e-fb75-4613-ba55-717b478af302\") " Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.642493 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.644453 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d8350e-fb75-4613-ba55-717b478af302-istiod-ca-cert" pod "d5d8350e-fb75-4613-ba55-717b478af302" (UID: "d5d8350e-fb75-4613-ba55-717b478af302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.644760 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-certs" pod "d5d8350e-fb75-4613-ba55-717b478af302" (UID: "d5d8350e-fb75-4613-ba55-717b478af302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.645170 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-data" pod "d5d8350e-fb75-4613-ba55-717b478af302" (UID: "d5d8350e-fb75-4613-ba55-717b478af302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:43.648343 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.645223 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-credential-socket" pod "d5d8350e-fb75-4613-ba55-717b478af302" (UID: "d5d8350e-fb75-4613-ba55-717b478af302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:43.649657 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.649368 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-envoy" pod "d5d8350e-fb75-4613-ba55-717b478af302" (UID: "d5d8350e-fb75-4613-ba55-717b478af302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:53:43.650214 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.650189 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5d8350e-fb75-4613-ba55-717b478af302-istio-podinfo" pod "d5d8350e-fb75-4613-ba55-717b478af302" (UID: "d5d8350e-fb75-4613-ba55-717b478af302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:53:43.650776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.650752 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-istio-token" pod "d5d8350e-fb75-4613-ba55-717b478af302" (UID: "d5d8350e-fb75-4613-ba55-717b478af302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:43.651135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.651118 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-kube-api-access-4mml6" pod "d5d8350e-fb75-4613-ba55-717b478af302" (UID: "d5d8350e-fb75-4613-ba55-717b478af302"). InnerVolumeSpecName "kube-api-access-4mml6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:53:43.743981 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.743951 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:43.743981 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.743979 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mml6\" (UniqueName: \"kubernetes.io/projected/d5d8350e-fb75-4613-ba55-717b478af302-kube-api-access-4mml6\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:43.744201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.743990 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d8350e-fb75-4613-ba55-717b478af302-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:43.744201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.744000 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:43.744201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.744010 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:43.744201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.744018 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:43.744201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.744025 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d8350e-fb75-4613-ba55-717b478af302-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:43.744201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.744032 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d8350e-fb75-4613-ba55-717b478af302-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:53:43.881463 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:43.881385 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:44.081622 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:44.081590 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:44.464413 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:44.464383 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:44.527724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:44.527687 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" event={"ID":"d5d8350e-fb75-4613-ba55-717b478af302","Type":"ContainerDied","Data":"9ede9e1f33e9d080cb49660b797a481db09cc2ba470c2e80ebdf12aa1cb4a26d"} Oct 30 12:53:44.527724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:44.527711 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t" Oct 30 12:53:44.528264 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:44.527753 5515 scope.go:122] "RemoveContainer" containerID="31f5a875c9b4aef3d216d583d392ceda891657a0e973b6c370f8e03d8b6cdaac" Oct 30 12:53:44.528264 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:44.527823 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:44.589291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:44.585899 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t"] Oct 30 12:53:44.599707 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:44.599670 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cc59b5af-openshift-default-6fcc8f67b5-8js2t"] Oct 30 12:53:44.667150 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:44.667112 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:46.001135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.001101 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d8350e-fb75-4613-ba55-717b478af302" path="/var/lib/kubelet/pods/d5d8350e-fb75-4613-ba55-717b478af302/volumes" Oct 30 12:53:46.238774 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.238744 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:46.619553 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.619518 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l"] Oct 30 12:53:46.620000 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.619988 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" Oct 30 12:53:46.620054 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.620002 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" Oct 30 12:53:46.620054 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.620014 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" Oct 30 12:53:46.620054 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.620020 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" Oct 30 12:53:46.620151 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.620102 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" Oct 30 12:53:46.620151 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.620110 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d8350e-fb75-4613-ba55-717b478af302" containerName="istio-proxy" Oct 30 12:53:46.624864 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.624847 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:46.642629 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.642605 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7a8b03\"" Oct 30 12:53:46.651993 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.651957 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:46.658634 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.658606 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l"] Oct 30 12:53:46.794644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.794603 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdkk\" (UniqueName: \"kubernetes.io/projected/75668c46-4685-4d6e-8a91-cc8f21364998-kube-api-access-5jdkk\") pod \"payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l\" (UID: \"75668c46-4685-4d6e-8a91-cc8f21364998\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:46.794832 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.794719 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75668c46-4685-4d6e-8a91-cc8f21364998-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l\" (UID: \"75668c46-4685-4d6e-8a91-cc8f21364998\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:46.895995 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.895964 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdkk\" (UniqueName: \"kubernetes.io/projected/75668c46-4685-4d6e-8a91-cc8f21364998-kube-api-access-5jdkk\") pod \"payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l\" (UID: \"75668c46-4685-4d6e-8a91-cc8f21364998\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:46.896151 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.896043 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75668c46-4685-4d6e-8a91-cc8f21364998-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l\" (UID: \"75668c46-4685-4d6e-8a91-cc8f21364998\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:46.896757 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.896735 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75668c46-4685-4d6e-8a91-cc8f21364998-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l\" (UID: \"75668c46-4685-4d6e-8a91-cc8f21364998\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:46.919690 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.919658 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdkk\" (UniqueName: \"kubernetes.io/projected/75668c46-4685-4d6e-8a91-cc8f21364998-kube-api-access-5jdkk\") pod \"payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l\" (UID: \"75668c46-4685-4d6e-8a91-cc8f21364998\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:46.937630 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.937600 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:46.961449 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:46.961418 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:47.236996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.236959 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l"] Oct 30 12:53:47.238026 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:47.238000 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75668c46_4685_4d6e_8a91_cc8f21364998.slice/crio-b0845ad473bb7b7d95d2deb647f2110bd837828df4af1e4b803201816a466d13 WatchSource:0}: Error finding container b0845ad473bb7b7d95d2deb647f2110bd837828df4af1e4b803201816a466d13: Status 404 returned error can't find the container with id b0845ad473bb7b7d95d2deb647f2110bd837828df4af1e4b803201816a466d13 Oct 30 12:53:47.540214 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.540130 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:47.545919 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.545890 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" event={"ID":"75668c46-4685-4d6e-8a91-cc8f21364998","Type":"ContainerStarted","Data":"a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530"} Oct 30 12:53:47.546028 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.545952 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:47.546028 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.545968 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" event={"ID":"75668c46-4685-4d6e-8a91-cc8f21364998","Type":"ContainerStarted","Data":"b0845ad473bb7b7d95d2deb647f2110bd837828df4af1e4b803201816a466d13"} Oct 30 12:53:47.585949 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.585890 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" podStartSLOduration=1.585877886 podStartE2EDuration="1.585877886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:47.585556838 +0000 UTC m=+1245.712357749" watchObservedRunningTime="2026-10-30 12:53:47.585877886 +0000 UTC m=+1245.712678784" Oct 30 12:53:47.722711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.722681 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg"] Oct 30 12:53:47.726373 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.726352 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:47.746506 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.746477 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg"] Oct 30 12:53:47.748915 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.748889 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:47.757399 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.757367 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-bb86fd-dockercfg-5jm9w\"" Oct 30 12:53:47.757524 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.757471 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-bb86fd\"" Oct 30 12:53:47.913649 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.913617 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg\" (UID: \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:47.913860 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:47.913840 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmhn\" (UniqueName: \"kubernetes.io/projected/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-kube-api-access-5xmhn\") pod \"payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg\" (UID: \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:48.010734 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.010703 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:48.014996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.014970 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmhn\" (UniqueName: \"kubernetes.io/projected/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-kube-api-access-5xmhn\") pod \"payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg\" (UID: \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:48.015119 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.015017 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg\" (UID: \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:48.015759 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.015741 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg\" (UID: \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:48.038023 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.037995 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmhn\" (UniqueName: \"kubernetes.io/projected/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-kube-api-access-5xmhn\") pod \"payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg\" (UID: \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:48.040590 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.040571 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:48.156292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.148609 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:53:48.216703 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.216613 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:48.355115 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.355052 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg"] Oct 30 12:53:48.357955 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:48.357923 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b6b991_deb5_44d6_a8e2_25af9eb6e6b0.slice/crio-bb7f3bbad300f87ee340767389cae7f6dae34f5bc3267fd53ebf22bb1692970f WatchSource:0}: Error finding container bb7f3bbad300f87ee340767389cae7f6dae34f5bc3267fd53ebf22bb1692970f: Status 404 returned error can't find the container with id bb7f3bbad300f87ee340767389cae7f6dae34f5bc3267fd53ebf22bb1692970f Oct 30 12:53:48.553409 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.553314 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" event={"ID":"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0","Type":"ContainerStarted","Data":"035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b"} Oct 30 12:53:48.553565 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.553464 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:48.553565 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.553494 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" event={"ID":"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0","Type":"ContainerStarted","Data":"bb7f3bbad300f87ee340767389cae7f6dae34f5bc3267fd53ebf22bb1692970f"} Oct 30 12:53:48.592316 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.592245 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" podStartSLOduration=1.5922302149999998 podStartE2EDuration="1.592230215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:48.591178255 +0000 UTC m=+1246.717979154" watchObservedRunningTime="2026-10-30 12:53:48.592230215 +0000 UTC m=+1246.719031113" Oct 30 12:53:48.729666 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.729635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:48.949834 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:48.949800 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:49.176660 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:49.176629 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:49.986224 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:49.986183 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:50.259878 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:50.259796 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:50.480694 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:50.480648 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:50.540639 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:50.540573 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 12:53:50.690562 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:50.690531 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:52.081968 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:52.081931 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:53.524650 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:53.524619 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 12:53:53.593658 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:53.593627 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:54.222394 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:54.222360 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:54.303571 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:54.303534 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:53:54.721063 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:54.721015 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:54.946501 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:54.946457 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:55.164836 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:55.164782 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:55.669378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:55.669340 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:56.946020 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:56.945993 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:57.546547 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.546510 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:57.644292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.641549 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455"] Oct 30 12:53:57.655018 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.654991 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:53:57.659657 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.659626 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8tn\" (UniqueName: \"kubernetes.io/projected/ad4c9f3d-fa30-4061-9ce5-9e557722441f-kube-api-access-sc8tn\") pod \"payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455\" (UID: \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:53:57.659791 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.659661 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c9f3d-fa30-4061-9ce5-9e557722441f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455\" (UID: \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:53:57.676455 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.676426 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455"] Oct 30 12:53:57.684896 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.684546 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-6ff0e6-dockercfg-c9588\"" Oct 30 12:53:57.731356 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.731321 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529"] Oct 30 12:53:57.786192 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.786158 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:53:57.813203 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.812084 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-6ff0e6\"" Oct 30 12:53:57.813950 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.813875 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:57.831293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.814232 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529"] Oct 30 12:53:57.847292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.845588 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8tn\" (UniqueName: \"kubernetes.io/projected/ad4c9f3d-fa30-4061-9ce5-9e557722441f-kube-api-access-sc8tn\") pod \"payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455\" (UID: \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:53:57.847292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.845651 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c9f3d-fa30-4061-9ce5-9e557722441f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455\" (UID: \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:53:57.863939 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.847844 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c9f3d-fa30-4061-9ce5-9e557722441f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455\" (UID: \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:53:57.878094 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.876121 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8tn\" (UniqueName: \"kubernetes.io/projected/ad4c9f3d-fa30-4061-9ce5-9e557722441f-kube-api-access-sc8tn\") pod \"payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455\" (UID: \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:53:57.946875 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.946831 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529\" (UID: \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:53:57.947473 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.947443 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvpdz\" (UniqueName: \"kubernetes.io/projected/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-kube-api-access-tvpdz\") pod \"payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529\" (UID: \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:53:57.973744 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:57.973716 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:53:58.052422 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.051756 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529\" (UID: \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:53:58.052422 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.051798 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvpdz\" (UniqueName: \"kubernetes.io/projected/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-kube-api-access-tvpdz\") pod \"payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529\" (UID: \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:53:58.052695 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.052564 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529\" (UID: \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:53:58.077052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.076978 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvpdz\" (UniqueName: \"kubernetes.io/projected/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-kube-api-access-tvpdz\") pod \"payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529\" (UID: \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:53:58.174209 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.174176 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:53:58.330341 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.329679 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455"] Oct 30 12:53:58.352248 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:58.352202 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4c9f3d_fa30_4061_9ce5_9e557722441f.slice/crio-ed7bc46686afc2f2d0be692d60a75deec10aad37bee1427560c6672b2c90aaa5 WatchSource:0}: Error finding container ed7bc46686afc2f2d0be692d60a75deec10aad37bee1427560c6672b2c90aaa5: Status 404 returned error can't find the container with id ed7bc46686afc2f2d0be692d60a75deec10aad37bee1427560c6672b2c90aaa5 Oct 30 12:53:58.531291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.527610 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529"] Oct 30 12:53:58.547516 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:53:58.546432 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69244e3_72b4_4d7d_b277_1ddd0c0e9de6.slice/crio-0e14a59c48e771a8df6fb5afb557f5eef94fc14d023287ba0a87ed04780a8732 WatchSource:0}: Error finding container 0e14a59c48e771a8df6fb5afb557f5eef94fc14d023287ba0a87ed04780a8732: Status 404 returned error can't find the container with id 0e14a59c48e771a8df6fb5afb557f5eef94fc14d023287ba0a87ed04780a8732 Oct 30 12:53:58.559794 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.556979 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 12:53:58.656515 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.656476 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" event={"ID":"ad4c9f3d-fa30-4061-9ce5-9e557722441f","Type":"ContainerStarted","Data":"c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520"} Oct 30 12:53:58.656702 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.656567 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" event={"ID":"ad4c9f3d-fa30-4061-9ce5-9e557722441f","Type":"ContainerStarted","Data":"ed7bc46686afc2f2d0be692d60a75deec10aad37bee1427560c6672b2c90aaa5"} Oct 30 12:53:58.658111 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.658085 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" event={"ID":"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6","Type":"ContainerStarted","Data":"0e14a59c48e771a8df6fb5afb557f5eef94fc14d023287ba0a87ed04780a8732"} Oct 30 12:53:58.715292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:58.698174 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" podStartSLOduration=1.698156335 podStartE2EDuration="1.698156335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:58.697866692 +0000 UTC m=+1256.824667592" watchObservedRunningTime="2026-10-30 12:53:58.698156335 +0000 UTC m=+1256.824957238" Oct 30 12:53:58.987587 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:53:58.987553 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:53:59.466284 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:59.466199 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:59.560871 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:59.560842 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 12:53:59.665543 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:59.665489 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" event={"ID":"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6","Type":"ContainerStarted","Data":"f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6"} Oct 30 12:53:59.665736 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:59.665610 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:53:59.665964 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:59.665942 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:53:59.752664 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:59.752546 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" podStartSLOduration=2.7525239839999998 podStartE2EDuration="2.752523984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:59.714522147 +0000 UTC m=+1257.841323047" watchObservedRunningTime="2026-10-30 12:53:59.752523984 +0000 UTC m=+1257.879324884" Oct 30 12:53:59.876930 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:53:59.875548 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:54:00.713802 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:00.713764 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:01.337042 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:01.336881 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:01.541671 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:01.541635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:02.510151 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:02.510053 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:03.086388 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.086350 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r"] Oct 30 12:54:03.091351 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.091326 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.110926 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.110803 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-6ff0e6-serving-cert\"" Oct 30 12:54:03.126387 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.126351 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r"] Oct 30 12:54:03.181595 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.181564 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:03.250183 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.250142 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-metrics-certs\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.250379 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.250199 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a4146d-bbb6-48c8-b47e-4372fdc387cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.250379 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.250310 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8jb\" (UniqueName: \"kubernetes.io/projected/07a4146d-bbb6-48c8-b47e-4372fdc387cc-kube-api-access-zb8jb\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.250456 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.250408 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.351545 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.351460 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.351724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.351584 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-metrics-certs\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.351724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.351613 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a4146d-bbb6-48c8-b47e-4372fdc387cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.351724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.351658 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8jb\" (UniqueName: \"kubernetes.io/projected/07a4146d-bbb6-48c8-b47e-4372fdc387cc-kube-api-access-zb8jb\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.352489 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.352461 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a4146d-bbb6-48c8-b47e-4372fdc387cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.354052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.354031 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.354221 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.354197 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-metrics-certs\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.377053 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.377020 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8jb\" (UniqueName: \"kubernetes.io/projected/07a4146d-bbb6-48c8-b47e-4372fdc387cc-kube-api-access-zb8jb\") pod \"maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.404737 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.404707 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:03.407852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.407821 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:03.617090 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.617003 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:03.717390 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.717353 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r"] Oct 30 12:54:03.929547 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:03.929508 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:04.173024 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:04.172991 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:04.703320 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:04.703287 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" event={"ID":"07a4146d-bbb6-48c8-b47e-4372fdc387cc","Type":"ContainerStarted","Data":"c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c"} Oct 30 12:54:04.703774 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:04.703345 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" event={"ID":"07a4146d-bbb6-48c8-b47e-4372fdc387cc","Type":"ContainerStarted","Data":"e5562d0f4226253b976a3df92d29c4001be71830b2baee04cd54730ef05281e9"} Oct 30 12:54:04.703774 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:04.703462 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:04.753278 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:04.753210 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" podStartSLOduration=1.3935391529999999 podStartE2EDuration="1.75319064s" totalImagesPullingTime="359.651487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:04.75099875 +0000 UTC m=+1262.877799647" watchObservedRunningTime="2026-10-30 12:54:04.75319064 +0000 UTC m=+1262.879991538" Oct 30 12:54:05.320305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.320240 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:05.710673 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.710641 5515 generic.go:356] "Generic (PLEG): container finished" podID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerID="983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c" exitCode=0 Oct 30 12:54:05.711100 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.710716 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" event={"ID":"52f163fa-56a6-411a-802b-d421d7b68e6e","Type":"ContainerDied","Data":"983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c"} Oct 30 12:54:05.711184 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.711166 5515 scope.go:122] "RemoveContainer" containerID="983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c" Oct 30 12:54:05.712568 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.712546 5515 generic.go:356] "Generic (PLEG): container finished" podID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerID="442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d" exitCode=0 Oct 30 12:54:05.712702 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.712570 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" event={"ID":"a7d51232-79de-44ac-a895-4553f87ed4e8","Type":"ContainerDied","Data":"442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d"} Oct 30 12:54:05.712702 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.712543 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:54:05.712702 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.712692 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:54:05.712879 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.712727 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:54:05.712939 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.712892 5515 scope.go:122] "RemoveContainer" containerID="442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d" Oct 30 12:54:05.714123 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.714092 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:54:05.714222 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.714155 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:54:05.714222 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:05.714183 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:54:06.745291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:06.744736 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" event={"ID":"52f163fa-56a6-411a-802b-d421d7b68e6e","Type":"ContainerStarted","Data":"4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5"} Oct 30 12:54:06.748705 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:06.748443 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" event={"ID":"a7d51232-79de-44ac-a895-4553f87ed4e8","Type":"ContainerStarted","Data":"9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a"} Oct 30 12:54:06.974682 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:06.974643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:07.017352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:07.016569 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:54:07.017352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:07.016755 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:54:07.017352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:07.016801 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 12:54:07.221765 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:07.221713 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:07.249982 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:07.249910 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:54:07.250924 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:07.250897 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:54:07.251085 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:07.250953 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:54:08.016736 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:08.016696 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:54:08.017384 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:08.017348 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 12:54:08.049288 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:08.049249 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:54:08.119298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:08.116240 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:08.264293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:08.253983 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:54:08.264293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:08.254056 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:54:08.327349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:08.327098 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:08.582143 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:08.582040 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:09.016809 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:09.016763 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:54:09.018135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:09.018098 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 12:54:09.250789 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:09.250748 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:54:09.250996 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:09.250804 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:54:10.035293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:10.017059 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:54:10.035293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:10.017120 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 12:54:10.181796 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:10.181755 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:10.250920 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:10.250878 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:54:10.251106 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:10.250932 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:54:10.470433 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:10.470399 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:54:10.674899 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:10.674648 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 12:54:10.674899 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:10.674720 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 12:54:10.752556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:10.752473 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 12:54:11.018823 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:11.018723 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:54:11.019101 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:11.019071 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 12:54:11.250783 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:11.250745 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 12:54:11.251173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:11.250800 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 12:54:11.515292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:11.514381 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:54:11.843963 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:11.843877 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:12.059214 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:12.059130 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:54:12.061135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:12.061111 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 12:54:12.254196 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:12.254171 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:54:12.255143 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:12.255124 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 12:54:12.718462 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:12.718431 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:54:13.379959 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:13.379914 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:13.668581 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:13.668551 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:13.990771 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:13.990689 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:14.976642 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:14.976613 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:16.571147 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:16.571113 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:18.098285 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:18.098242 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:18.310032 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:18.309966 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:19.872722 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:19.872689 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:20.079484 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:20.079453 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:20.545885 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:20.545856 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:20.757604 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:20.757547 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:21.545701 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:21.545669 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:22.281814 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:22.281779 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:54:22.999013 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:22.998977 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:24.675574 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:24.675535 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:26.257134 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:26.257086 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:27.708279 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:27.708224 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:27.911260 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:27.911224 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:28.987413 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:54:28.987378 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:54:29.265662 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:29.265551 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:30.799534 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:30.799499 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:32.837490 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:32.837458 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:34.173656 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.173624 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:34.820420 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.820388 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9"] Oct 30 12:54:34.823403 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.823381 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:34.855813 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.855784 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"openshift-service-ca.crt\"" Oct 30 12:54:34.856003 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.855792 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"premium-w3-551dd1-kserve-self-signed-certs\"" Oct 30 12:54:34.856003 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.855830 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"default-dockercfg-mxsm5\"" Oct 30 12:54:34.856169 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.855835 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"kube-root-ca.crt\"" Oct 30 12:54:34.859061 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.859036 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9"] Oct 30 12:54:34.967650 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.967616 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:34.986436 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.986408 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tls-certs\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:34.986556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.986513 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-home\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:34.986732 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.986615 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htz85\" (UniqueName: \"kubernetes.io/projected/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-kube-api-access-htz85\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:34.986822 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.986729 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-dshm\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:34.986822 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.986753 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tmp-dir\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:34.986822 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:34.986791 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-model-cache\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.087711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.087627 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tls-certs\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.087711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.087686 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-home\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.087936 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.087724 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htz85\" (UniqueName: \"kubernetes.io/projected/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-kube-api-access-htz85\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.087936 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.087784 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-dshm\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.087936 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.087805 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tmp-dir\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.087936 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.087838 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-model-cache\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.088173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.088149 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-home\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.088246 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.088197 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tmp-dir\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.088406 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.088385 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-model-cache\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.089983 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.089960 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-dshm\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.090235 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.090218 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tls-certs\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.111120 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.111093 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htz85\" (UniqueName: \"kubernetes.io/projected/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-kube-api-access-htz85\") pod \"premium-w3-551dd1-kserve-86b4965b75-tjms9\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.135828 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.135806 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:35.428965 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.428938 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9"] Oct 30 12:54:35.429903 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:54:35.429866 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2516ca_0565_4a71_ba56_48e1b1aa3fc1.slice/crio-7d445b2518beaa293b54b87cfacbf4098a5fcfd5a9fc58949415bf3e0c85aae4 WatchSource:0}: Error finding container 7d445b2518beaa293b54b87cfacbf4098a5fcfd5a9fc58949415bf3e0c85aae4: Status 404 returned error can't find the container with id 7d445b2518beaa293b54b87cfacbf4098a5fcfd5a9fc58949415bf3e0c85aae4 Oct 30 12:54:35.769758 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.769672 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:35.914863 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.914827 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" event={"ID":"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1","Type":"ContainerStarted","Data":"96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6"} Oct 30 12:54:35.915054 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.914873 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" event={"ID":"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1","Type":"ContainerStarted","Data":"7d445b2518beaa293b54b87cfacbf4098a5fcfd5a9fc58949415bf3e0c85aae4"} Oct 30 12:54:35.958624 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:35.958566 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podStartSLOduration=1.958553064 podStartE2EDuration="1.958553064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:35.956847827 +0000 UTC m=+1294.083648726" watchObservedRunningTime="2026-10-30 12:54:35.958553064 +0000 UTC m=+1294.085353962" Oct 30 12:54:36.136920 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:36.136889 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:37.376055 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:37.376018 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:38.194445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:38.194411 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:38.548875 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:38.548787 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:54:38.835251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:38.835161 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:40.386508 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:40.386476 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:41.987769 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:41.987737 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:43.463398 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:43.463356 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:43.685627 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:43.685591 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:45.085676 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:45.085633 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:45.136862 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:45.136827 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:45.149520 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:45.149492 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:45.984027 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:45.983997 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:54:46.153242 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:46.153184 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:46.351660 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:46.351578 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:46.581321 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:46.581289 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:48.250966 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:48.250934 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:49.773590 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:49.773555 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:51.399216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:51.399176 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:52.978093 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:52.978063 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:54.563830 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.563795 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:54.774136 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.774094 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb"] Oct 30 12:54:54.777106 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.777084 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.792771 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.792746 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"kube-root-ca.crt\"" Oct 30 12:54:54.794353 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.794328 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-model-cache\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.794456 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.794374 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bf3958-6eff-4b10-9d14-9ae43712b46d-tls-certs\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.794456 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.794443 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5ddp\" (UniqueName: \"kubernetes.io/projected/72bf3958-6eff-4b10-9d14-9ae43712b46d-kube-api-access-z5ddp\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.794532 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.794502 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-home\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.794599 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.794578 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-dshm\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.794738 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.794723 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-tmp-dir\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.804344 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.804309 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"openshift-service-ca.crt\"" Oct 30 12:54:54.804473 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.804413 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"default-dockercfg-5xdjs\"" Oct 30 12:54:54.806621 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.806597 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb"] Oct 30 12:54:54.892667 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.892637 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"premium-w2-bb86fd-kserve-self-signed-certs\"" Oct 30 12:54:54.895771 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.895748 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-tmp-dir\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.895858 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.895785 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-model-cache\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.895858 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.895820 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bf3958-6eff-4b10-9d14-9ae43712b46d-tls-certs\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.895858 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.895847 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5ddp\" (UniqueName: \"kubernetes.io/projected/72bf3958-6eff-4b10-9d14-9ae43712b46d-kube-api-access-z5ddp\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.895965 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.895908 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-home\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.895965 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.895939 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-dshm\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.896221 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.896205 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-tmp-dir\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.896331 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.896262 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-model-cache\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.896447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.896425 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-home\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.898156 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.898138 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-dshm\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.898445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.898429 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bf3958-6eff-4b10-9d14-9ae43712b46d-tls-certs\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.918927 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.918904 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5ddp\" (UniqueName: \"kubernetes.io/projected/72bf3958-6eff-4b10-9d14-9ae43712b46d-kube-api-access-z5ddp\") pod \"premium-w2-bb86fd-kserve-669fd7db8d-zxlrb\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:54.923689 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:54.923667 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:55.091705 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:55.091667 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:55.390109 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:55.390083 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb"] Oct 30 12:54:55.391087 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:54:55.391062 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72bf3958_6eff_4b10_9d14_9ae43712b46d.slice/crio-30c6031b2e2a2ca77c939799464f084126b66c66e87bfdfdc008bb7dd6f1c17f WatchSource:0}: Error finding container 30c6031b2e2a2ca77c939799464f084126b66c66e87bfdfdc008bb7dd6f1c17f: Status 404 returned error can't find the container with id 30c6031b2e2a2ca77c939799464f084126b66c66e87bfdfdc008bb7dd6f1c17f Oct 30 12:54:56.028331 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:56.028294 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" event={"ID":"72bf3958-6eff-4b10-9d14-9ae43712b46d","Type":"ContainerStarted","Data":"e1f907b9dec7518c0c3220d868372b5a30a0ee3ddc80a2b2b941e12907506d67"} Oct 30 12:54:56.028742 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:56.028338 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" event={"ID":"72bf3958-6eff-4b10-9d14-9ae43712b46d","Type":"ContainerStarted","Data":"30c6031b2e2a2ca77c939799464f084126b66c66e87bfdfdc008bb7dd6f1c17f"} Oct 30 12:54:56.071099 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:56.071044 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podStartSLOduration=2.071029402 podStartE2EDuration="2.071029402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:56.069670733 +0000 UTC m=+1314.196471630" watchObservedRunningTime="2026-10-30 12:54:56.071029402 +0000 UTC m=+1314.197830300" Oct 30 12:54:56.092181 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:56.092136 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:54:56.182709 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:56.182662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:57.741622 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:57.741589 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:57.922682 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:57.922641 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:54:58.989057 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:54:58.989012 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:54:59.540653 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:59.540614 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:59.751068 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:54:59.751033 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:01.058972 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:01.058930 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:02.583043 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:02.583006 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:04.171981 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:04.171947 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:05.091947 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:05.091906 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:55:05.104858 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:05.104827 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:55:05.773697 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:05.773664 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:06.096119 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:06.096031 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:55:07.286747 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:07.286701 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:08.296751 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:08.296710 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:08.504055 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:08.504022 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:09.098771 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:09.098736 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:10.875904 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:10.875871 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:12.512418 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:12.512384 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:14.381973 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:14.381940 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:15.629918 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:15.629876 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:55:15.949515 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:15.949486 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:17.588634 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:17.588581 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:19.182558 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:19.182514 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:20.684641 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:20.684603 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:22.222536 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:22.222506 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:23.681571 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:23.681542 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:25.282606 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:25.282566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:26.885780 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:26.885748 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:28.674952 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:28.674921 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:28.987014 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:55:28.986939 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:55:30.080622 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:30.080595 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:31.673674 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:31.673641 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:33.274760 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:33.274720 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:34.413891 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:34.413854 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:55:34.884978 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:34.884943 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:36.489998 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:36.489969 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:38.083686 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:38.083650 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:39.668754 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:39.668719 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:41.437529 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:41.437490 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:43.441731 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:43.441646 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:45.258502 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:45.258469 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:46.782579 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:46.782530 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:48.575280 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:48.575224 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:50.089162 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.089125 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:50.132632 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.132598 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z"] Oct 30 12:55:50.137407 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.137380 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.169344 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.169220 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z"] Oct 30 12:55:50.178289 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.178249 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 12:55:50.178478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.178337 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\"" Oct 30 12:55:50.212868 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.212818 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-100c91cd-rtt5z\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.213064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.213022 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sll9v\" (UniqueName: \"kubernetes.io/projected/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-kube-api-access-sll9v\") pod \"maas-api-revoke-sub-100c91cd-rtt5z\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.213064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.213057 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-100c91cd-rtt5z\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.361502 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.359862 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-100c91cd-rtt5z\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.361502 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.360118 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sll9v\" (UniqueName: \"kubernetes.io/projected/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-kube-api-access-sll9v\") pod \"maas-api-revoke-sub-100c91cd-rtt5z\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.361502 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.360167 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-100c91cd-rtt5z\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.361502 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.361137 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-100c91cd-rtt5z\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.363426 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.363402 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-100c91cd-rtt5z\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.383841 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.383807 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sll9v\" (UniqueName: \"kubernetes.io/projected/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-kube-api-access-sll9v\") pod \"maas-api-revoke-sub-100c91cd-rtt5z\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.461514 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.459521 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:50.771506 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:50.771471 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z"] Oct 30 12:55:50.772309 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:55:50.772261 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09e9acd_faec_4cfb_b94c_1e7b04e7fa50.slice/crio-7ae894ac0097355227d1a4278ed393050619331a611f38d4fa0caf56e07f04ed WatchSource:0}: Error finding container 7ae894ac0097355227d1a4278ed393050619331a611f38d4fa0caf56e07f04ed: Status 404 returned error can't find the container with id 7ae894ac0097355227d1a4278ed393050619331a611f38d4fa0caf56e07f04ed Oct 30 12:55:51.371129 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:51.371041 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" event={"ID":"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50","Type":"ContainerStarted","Data":"56a8c55b70c2b3f7ab9b9e8a17bc7c37c407812f15497d28b0546098ccfab8b8"} Oct 30 12:55:51.371129 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:51.371091 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" event={"ID":"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50","Type":"ContainerStarted","Data":"7ae894ac0097355227d1a4278ed393050619331a611f38d4fa0caf56e07f04ed"} Oct 30 12:55:51.412387 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:51.412335 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" podStartSLOduration=1.412322971 podStartE2EDuration="1.412322971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:51.41068402 +0000 UTC m=+1369.537484919" watchObservedRunningTime="2026-10-30 12:55:51.412322971 +0000 UTC m=+1369.539123869" Oct 30 12:55:51.677546 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:51.677512 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:52.377657 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:52.377624 5515 generic.go:356] "Generic (PLEG): container finished" podID="b09e9acd-faec-4cfb-b94c-1e7b04e7fa50" containerID="56a8c55b70c2b3f7ab9b9e8a17bc7c37c407812f15497d28b0546098ccfab8b8" exitCode=0 Oct 30 12:55:52.378059 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:52.377692 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" event={"ID":"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50","Type":"ContainerDied","Data":"56a8c55b70c2b3f7ab9b9e8a17bc7c37c407812f15497d28b0546098ccfab8b8"} Oct 30 12:55:53.180109 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.180050 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:53.525414 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.525386 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:53.694417 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.694325 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-openshift-ca\") pod \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " Oct 30 12:55:53.694650 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.694554 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-xks-ca\") pod \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " Oct 30 12:55:53.694650 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.694616 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-kube-api-access-sll9v\" (UniqueName: \"kubernetes.io/projected/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-kube-api-access-sll9v\") pod \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\" (UID: \"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50\") " Oct 30 12:55:53.694774 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.694669 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-openshift-ca" pod "b09e9acd-faec-4cfb-b94c-1e7b04e7fa50" (UID: "b09e9acd-faec-4cfb-b94c-1e7b04e7fa50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:55:53.694973 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.694955 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-openshift-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:55:53.696591 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.696568 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-xks-ca" pod "b09e9acd-faec-4cfb-b94c-1e7b04e7fa50" (UID: "b09e9acd-faec-4cfb-b94c-1e7b04e7fa50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:55:53.696712 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.696687 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-kube-api-access-sll9v" pod "b09e9acd-faec-4cfb-b94c-1e7b04e7fa50" (UID: "b09e9acd-faec-4cfb-b94c-1e7b04e7fa50"). InnerVolumeSpecName "kube-api-access-sll9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:53.796132 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.796098 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-maas-api-xks-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:55:53.796132 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:53.796129 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sll9v\" (UniqueName: \"kubernetes.io/projected/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50-kube-api-access-sll9v\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:55:54.390441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:54.390410 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" Oct 30 12:55:54.390441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:54.390421 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z" event={"ID":"b09e9acd-faec-4cfb-b94c-1e7b04e7fa50","Type":"ContainerDied","Data":"7ae894ac0097355227d1a4278ed393050619331a611f38d4fa0caf56e07f04ed"} Oct 30 12:55:54.390635 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:54.390464 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ae894ac0097355227d1a4278ed393050619331a611f38d4fa0caf56e07f04ed" Oct 30 12:55:54.992833 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:54.992799 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:56.681173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:56.681133 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:58.301483 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:58.301449 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:58.997612 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:55:58.997578 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:55:59.978612 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:55:59.978574 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:00.592685 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:00.592652 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:00.811351 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:00.811314 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:01.847169 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:01.847128 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:02.117216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.117102 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:02.279549 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.279514 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d654b97b9-6dss4"] Oct 30 12:56:02.280031 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.280018 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b09e9acd-faec-4cfb-b94c-1e7b04e7fa50" containerName="revoke-keys" Oct 30 12:56:02.280078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.280033 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09e9acd-faec-4cfb-b94c-1e7b04e7fa50" containerName="revoke-keys" Oct 30 12:56:02.280144 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.280134 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09e9acd-faec-4cfb-b94c-1e7b04e7fa50" containerName="revoke-keys" Oct 30 12:56:02.283686 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.283670 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.301003 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.300977 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dc5c44d3e94a\"" Oct 30 12:56:02.313280 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.313234 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 12:56:02.313439 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.313311 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-69b689882f95\"" Oct 30 12:56:02.313518 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.313498 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 12:56:02.316194 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.316167 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d654b97b9-6dss4"] Oct 30 12:56:02.397951 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.397907 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-plugins-config-volume\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.398387 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.398354 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-routing-overlay-volume\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.398506 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.398403 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfbn\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-kube-api-access-fzfbn\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.398556 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.398526 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-provider-credentials\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.403044 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.403013 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vsw6g\"" Oct 30 12:56:02.403168 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.403106 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 12:56:02.403211 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.403177 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 12:56:02.499791 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.499758 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-provider-credentials\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.500064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.499828 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-plugins-config-volume\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.500064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.499925 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-routing-overlay-volume\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.500064 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.499952 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfbn\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-kube-api-access-fzfbn\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.500928 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.500892 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-routing-overlay-volume\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.501235 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.501209 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-plugins-config-volume\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.502719 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.502696 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-provider-credentials\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.523728 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.523692 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfbn\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-kube-api-access-fzfbn\") pod \"payload-processing-external-model-d654b97b9-6dss4\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.597389 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.597350 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:02.898045 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:02.898012 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d654b97b9-6dss4"] Oct 30 12:56:02.900069 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:56:02.900014 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42f0042_dc03_4cd7_83cc_30da4513ff7f.slice/crio-1a7dfacc03e9897cdf96412f175dee4d5f56c6c16850968eca844ed6f9a41b43 WatchSource:0}: Error finding container 1a7dfacc03e9897cdf96412f175dee4d5f56c6c16850968eca844ed6f9a41b43: Status 404 returned error can't find the container with id 1a7dfacc03e9897cdf96412f175dee4d5f56c6c16850968eca844ed6f9a41b43 Oct 30 12:56:03.449676 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:03.449618 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" event={"ID":"b42f0042-dc03-4cd7-83cc-30da4513ff7f","Type":"ContainerStarted","Data":"2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd"} Oct 30 12:56:03.449676 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:03.449684 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" event={"ID":"b42f0042-dc03-4cd7-83cc-30da4513ff7f","Type":"ContainerStarted","Data":"1a7dfacc03e9897cdf96412f175dee4d5f56c6c16850968eca844ed6f9a41b43"} Oct 30 12:56:03.450644 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:03.450619 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:03.499622 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:03.499543 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" podStartSLOduration=1.499521966 podStartE2EDuration="1.499521966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:03.497698342 +0000 UTC m=+1381.624499240" watchObservedRunningTime="2026-10-30 12:56:03.499521966 +0000 UTC m=+1381.626322866" Oct 30 12:56:03.510240 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:03.510210 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:05.258783 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:05.258750 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:05.280797 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:05.280765 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:56:06.995293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:06.994942 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj"] Oct 30 12:56:07.014293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.004860 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.028674 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.028647 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e652ca-openshift-default-dockercfg-rr2pm\"" Oct 30 12:56:07.045918 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.045882 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj"] Oct 30 12:56:07.062304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.061217 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.062304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.061263 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed628cdb-692c-4357-8461-4e8af43a1d5c-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.062304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.061347 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.062304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.061406 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.062304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.061441 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgtk\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-kube-api-access-8wgtk\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.062304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.061513 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.062304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.061596 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.062304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.061621 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.062304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.061663 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.165968 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.165917 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.166245 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.166221 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed628cdb-692c-4357-8461-4e8af43a1d5c-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.166488 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.166469 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.166645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.166630 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.166772 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.166754 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgtk\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-kube-api-access-8wgtk\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.166916 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.166902 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.167078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.167063 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.167186 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.167172 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.167339 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.167320 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.168802 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.168101 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.168802 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.168710 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed628cdb-692c-4357-8461-4e8af43a1d5c-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.168997 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.168971 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.169195 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.169174 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.169807 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.169787 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.171444 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.170619 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.173603 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.173579 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.224291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.211016 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgtk\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-kube-api-access-8wgtk\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.224291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.211517 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.335341 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.331486 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:07.491486 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.489621 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:07.872844 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.871405 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj"] Oct 30 12:56:07.880889 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.880099 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:56:07.880889 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.880198 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:56:07.880889 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:07.880247 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:56:08.587507 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:08.587465 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:08.594012 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:08.593970 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" event={"ID":"ed628cdb-692c-4357-8461-4e8af43a1d5c","Type":"ContainerStarted","Data":"aa74bb58d202ab9521bf686b9c3cd7796248f0bd5761b2861fdb3165beef11d3"} Oct 30 12:56:08.594298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:08.594252 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" event={"ID":"ed628cdb-692c-4357-8461-4e8af43a1d5c","Type":"ContainerStarted","Data":"c25c1d5f0a572ef3dbed6b402ea409d07f04364c19bdd6379cfb033951ae890b"} Oct 30 12:56:08.686289 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:08.685836 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" podStartSLOduration=2.685803013 podStartE2EDuration="2.685803013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:08.679072549 +0000 UTC m=+1386.805873448" watchObservedRunningTime="2026-10-30 12:56:08.685803013 +0000 UTC m=+1386.812603911" Oct 30 12:56:09.332362 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:09.332319 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:09.334217 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:09.334189 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:56:09.334363 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:09.334236 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:56:09.666884 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:09.666847 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:10.332788 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:10.332751 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:56:10.332966 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:10.332826 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:56:10.773311 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:10.773257 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:11.333399 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:11.333358 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:56:11.333584 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:11.333426 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:56:11.449791 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:11.449749 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:12.333261 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:12.333224 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:56:12.333679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:12.333294 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:56:13.314339 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:13.314209 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:13.333279 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:13.333234 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:56:13.333453 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:13.333299 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:56:14.333104 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:14.333070 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 12:56:14.333319 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:14.333148 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 12:56:14.481395 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:14.481357 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:15.219609 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:15.219575 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:15.337298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:15.337259 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:15.337688 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:15.337662 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:15.338407 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:15.338388 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 12:56:15.498062 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:15.497979 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:17.110312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:17.110263 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:19.117201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:19.117170 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:20.406773 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:20.406743 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:56:20.934359 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:20.931251 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d"] Oct 30 12:56:20.950249 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:20.949847 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d"] Oct 30 12:56:20.950249 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:20.949968 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:20.985311 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:20.985282 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-551dd1-openshift-default-dockercfg-mnncw\"" Oct 30 12:56:20.988134 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:20.988040 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:21.138282 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.138246 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.138463 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.138356 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.138463 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.138414 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8mt\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-kube-api-access-lk8mt\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.138541 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.138509 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.138576 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.138546 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.138622 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.138575 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.138708 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.138677 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.138754 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.138731 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.138836 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.138818 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240238 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240507 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240338 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240507 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240384 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240507 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240415 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240507 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240446 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240507 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240486 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240806 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240604 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240806 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240672 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240806 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240698 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240806 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240716 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8mt\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-kube-api-access-lk8mt\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.240806 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.240772 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.241175 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.241135 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.241308 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.241176 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.241417 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.241395 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.242965 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.242936 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.243050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.243025 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.289957 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.289921 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.290460 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.290438 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8mt\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-kube-api-access-lk8mt\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.585479 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.585387 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:21.936898 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.936865 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d"] Oct 30 12:56:21.937795 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:56:21.937770 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e39a4b_a64e_4689_8b46_15c4a3e2ce48.slice/crio-b1e1b04fd73b51f773527b80d77c9397ed6bf6bf16450b70a0cc054f6b5395e2 WatchSource:0}: Error finding container b1e1b04fd73b51f773527b80d77c9397ed6bf6bf16450b70a0cc054f6b5395e2: Status 404 returned error can't find the container with id b1e1b04fd73b51f773527b80d77c9397ed6bf6bf16450b70a0cc054f6b5395e2 Oct 30 12:56:21.940132 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.940100 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:56:21.940201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.940164 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:56:21.940201 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:21.940194 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:56:22.729420 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:22.729379 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" event={"ID":"57e39a4b-a64e-4689-8b46-15c4a3e2ce48","Type":"ContainerStarted","Data":"cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf"} Oct 30 12:56:22.729817 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:22.729439 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" event={"ID":"57e39a4b-a64e-4689-8b46-15c4a3e2ce48","Type":"ContainerStarted","Data":"b1e1b04fd73b51f773527b80d77c9397ed6bf6bf16450b70a0cc054f6b5395e2"} Oct 30 12:56:22.807646 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:22.807576 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" podStartSLOduration=2.807555561 podStartE2EDuration="2.807555561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:22.793569964 +0000 UTC m=+1400.920370863" watchObservedRunningTime="2026-10-30 12:56:22.807555561 +0000 UTC m=+1400.934356460" Oct 30 12:56:23.198882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:23.198817 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:23.586315 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:23.586223 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:23.587859 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:23.587825 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:56:23.587991 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:23.587887 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 12:56:24.586500 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:24.586468 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:56:24.586906 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:24.586519 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 12:56:24.987613 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:24.987574 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:25.586738 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:25.586707 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:56:25.587151 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:25.586759 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 12:56:26.586347 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:26.586315 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:56:26.586795 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:26.586373 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 12:56:26.978929 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:26.978898 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:27.586413 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:27.586373 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:56:27.586593 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:27.586440 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 12:56:28.586323 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:28.586253 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:56:28.586717 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:28.586347 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 12:56:28.988491 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:28.988439 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rd49z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:56:28.988807 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:28.988761 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:29.082089 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:56:29.082037 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:56:29.468135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:29.468095 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:29.590815 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:29.590785 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:29.591220 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:29.591179 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:29.591962 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:29.591938 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 12:56:31.899670 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:31.899621 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:32.151955 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:32.151866 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:33.270912 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.270875 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4"] Oct 30 12:56:33.274889 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.274856 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.305034 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.305006 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4"] Oct 30 12:56:33.357150 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.357115 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-provider-credentials\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.357349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.357162 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-routing-overlay-volume\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.357349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.357236 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngg8\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-kube-api-access-6ngg8\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.357437 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.357383 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-plugins-config-volume\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.458787 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.458757 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-provider-credentials\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.458989 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.458800 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-routing-overlay-volume\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.458989 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.458830 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngg8\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-kube-api-access-6ngg8\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.458989 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.458960 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-plugins-config-volume\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.459528 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.459503 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-routing-overlay-volume\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.459652 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.459620 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-plugins-config-volume\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.461352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.461335 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-provider-credentials\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.482710 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.482685 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngg8\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-kube-api-access-6ngg8\") pod \"payload-processing-external-model-cdc946f8b-psdq4\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.589031 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.588958 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:33.919864 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.919829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:33.941529 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:33.941493 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4"] Oct 30 12:56:34.838607 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:34.838573 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" event={"ID":"bb3c06e3-5e09-4c97-95d7-f0672c4cc246","Type":"ContainerStarted","Data":"d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a"} Oct 30 12:56:34.839044 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:34.838630 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" event={"ID":"bb3c06e3-5e09-4c97-95d7-f0672c4cc246","Type":"ContainerStarted","Data":"3f7841caa94f743645d1e981bb325e5e31a0e2f39fde51ed281eb7c1558a2b80"} Oct 30 12:56:34.839044 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:34.838725 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:34.882534 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:34.882483 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" podStartSLOduration=1.882470127 podStartE2EDuration="1.882470127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:34.880847613 +0000 UTC m=+1413.007648513" watchObservedRunningTime="2026-10-30 12:56:34.882470127 +0000 UTC m=+1413.009271024" Oct 30 12:56:35.508503 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:35.508470 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:35.640549 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:35.640518 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:56:36.726485 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:36.726449 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:37.531735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:37.531687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:39.647449 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:39.647410 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:41.127961 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:41.127901 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:42.630447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:42.630397 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:44.256776 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:44.256734 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:45.848286 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:45.848236 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:56:45.962788 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:45.962732 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:45.964355 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:45.964328 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d654b97b9-6dss4"] Oct 30 12:56:46.808432 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:46.808400 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:47.592779 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:47.592735 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:49.200874 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:49.200798 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:50.869179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:50.869146 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:50.964835 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:50.964795 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" podUID="b42f0042-dc03-4cd7-83cc-30da4513ff7f" containerName="payload-processing" containerID="cri-o://2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd" gracePeriod=25 Oct 30 12:56:51.220005 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.219982 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:51.225362 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.225340 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-plugins-config-volume\") pod \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " Oct 30 12:56:51.225688 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.225667 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-plugins-config-volume" pod "b42f0042-dc03-4cd7-83cc-30da4513ff7f" (UID: "b42f0042-dc03-4cd7-83cc-30da4513ff7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:56:51.326426 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.326393 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-provider-credentials\") pod \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " Oct 30 12:56:51.326592 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.326449 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-kube-api-access-fzfbn\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-kube-api-access-fzfbn\") pod \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " Oct 30 12:56:51.326592 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.326583 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-routing-overlay-volume\") pod \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\" (UID: \"b42f0042-dc03-4cd7-83cc-30da4513ff7f\") " Oct 30 12:56:51.326882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.326858 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:56:51.326999 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.326968 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-routing-overlay-volume" pod "b42f0042-dc03-4cd7-83cc-30da4513ff7f" (UID: "b42f0042-dc03-4cd7-83cc-30da4513ff7f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:56:51.328758 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.328730 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-kube-api-access-fzfbn" pod "b42f0042-dc03-4cd7-83cc-30da4513ff7f" (UID: "b42f0042-dc03-4cd7-83cc-30da4513ff7f"). InnerVolumeSpecName "kube-api-access-fzfbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:56:51.328758 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.328734 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-provider-credentials" pod "b42f0042-dc03-4cd7-83cc-30da4513ff7f" (UID: "b42f0042-dc03-4cd7-83cc-30da4513ff7f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:56:51.427768 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.427686 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzfbn\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-kube-api-access-fzfbn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:56:51.427768 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.427715 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b42f0042-dc03-4cd7-83cc-30da4513ff7f-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:56:51.427768 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.427726 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b42f0042-dc03-4cd7-83cc-30da4513ff7f-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:56:51.966321 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.966249 5515 generic.go:356] "Generic (PLEG): container finished" podID="b42f0042-dc03-4cd7-83cc-30da4513ff7f" containerID="2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd" exitCode=0 Oct 30 12:56:51.966321 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.966316 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" Oct 30 12:56:51.966813 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.966331 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" event={"ID":"b42f0042-dc03-4cd7-83cc-30da4513ff7f","Type":"ContainerDied","Data":"2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd"} Oct 30 12:56:51.966813 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.966383 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d654b97b9-6dss4" event={"ID":"b42f0042-dc03-4cd7-83cc-30da4513ff7f","Type":"ContainerDied","Data":"1a7dfacc03e9897cdf96412f175dee4d5f56c6c16850968eca844ed6f9a41b43"} Oct 30 12:56:51.966813 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.966404 5515 scope.go:122] "RemoveContainer" containerID="2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd" Oct 30 12:56:51.979662 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.979639 5515 scope.go:122] "RemoveContainer" containerID="2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd" Oct 30 12:56:51.980110 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:56:51.980079 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd\": container with ID starting with 2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd not found: ID does not exist" containerID="2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd" Oct 30 12:56:51.980191 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:51.980120 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd"} err="failed to get container status \"2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd\": rpc error: code = NotFound desc = could not find container \"2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd\": container with ID starting with 2e5c7c4c1c3e9b25cd2db27ce2c4af5044461d6c2d1672ff5081f32f816da1dd not found: ID does not exist" Oct 30 12:56:52.024042 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:52.024013 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d654b97b9-6dss4"] Oct 30 12:56:52.036104 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:52.036071 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d654b97b9-6dss4"] Oct 30 12:56:52.520240 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:52.520203 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:54.000645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:54.000614 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42f0042-dc03-4cd7-83cc-30da4513ff7f" path="/var/lib/kubelet/pods/b42f0042-dc03-4cd7-83cc-30da4513ff7f/volumes" Oct 30 12:56:54.254862 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:54.254784 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:54.516803 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:54.516710 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:55.790636 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.790607 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-947b6b675-9k2tf"] Oct 30 12:56:55.791128 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.791115 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b42f0042-dc03-4cd7-83cc-30da4513ff7f" containerName="payload-processing" Oct 30 12:56:55.791185 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.791129 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42f0042-dc03-4cd7-83cc-30da4513ff7f" containerName="payload-processing" Oct 30 12:56:55.791245 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.791235 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42f0042-dc03-4cd7-83cc-30da4513ff7f" containerName="payload-processing" Oct 30 12:56:55.796059 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.796038 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.814026 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.814001 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-be5ef9d8ba28\"" Oct 30 12:56:55.814249 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.814139 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-54e16ed42556\"" Oct 30 12:56:55.828946 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.828920 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-947b6b675-9k2tf"] Oct 30 12:56:55.897482 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.897448 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkct\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-kube-api-access-hbkct\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.897650 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.897520 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-routing-overlay-volume\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.897650 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.897595 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-plugins-config-volume\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.897724 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.897658 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-provider-credentials\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.988387 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.988354 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:55.998906 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.998879 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-provider-credentials\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.999050 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.998960 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkct\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-kube-api-access-hbkct\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.999115 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.999050 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-routing-overlay-volume\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.999115 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.999090 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-plugins-config-volume\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.999721 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.999700 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-routing-overlay-volume\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:55.999823 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:55.999804 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-plugins-config-volume\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:56.001471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:56.001451 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-provider-credentials\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:56.024953 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:56.024928 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkct\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-kube-api-access-hbkct\") pod \"payload-processing-external-model-947b6b675-9k2tf\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:56.109418 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:56.109330 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:56.410051 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:56.410016 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-947b6b675-9k2tf"] Oct 30 12:56:56.412156 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:56.412128 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:56:56.412712 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:56:56.412682 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f03984b_2100_4f15_821e_e7a15a7a41ca.slice/crio-53ddb709b4ac2a8dc7f589e3961fdf7476f920be5a20b1ee1b3b5e430bc9792d WatchSource:0}: Error finding container 53ddb709b4ac2a8dc7f589e3961fdf7476f920be5a20b1ee1b3b5e430bc9792d: Status 404 returned error can't find the container with id 53ddb709b4ac2a8dc7f589e3961fdf7476f920be5a20b1ee1b3b5e430bc9792d Oct 30 12:56:57.035882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:57.035848 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" event={"ID":"5f03984b-2100-4f15-821e-e7a15a7a41ca","Type":"ContainerStarted","Data":"98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163"} Oct 30 12:56:57.036436 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:57.035907 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" event={"ID":"5f03984b-2100-4f15-821e-e7a15a7a41ca","Type":"ContainerStarted","Data":"53ddb709b4ac2a8dc7f589e3961fdf7476f920be5a20b1ee1b3b5e430bc9792d"} Oct 30 12:56:57.036817 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:57.036788 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:56:57.081255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:57.081198 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" podStartSLOduration=2.081185412 podStartE2EDuration="2.081185412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:57.078680171 +0000 UTC m=+1435.205481069" watchObservedRunningTime="2026-10-30 12:56:57.081185412 +0000 UTC m=+1435.207986310" Oct 30 12:56:57.782927 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:57.782891 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:58.987038 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:56:58.987000 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:56:59.569502 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:56:59.569469 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:01.270354 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:01.270317 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:02.978095 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:02.978061 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:03.532972 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.532935 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl"] Oct 30 12:57:03.539229 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.539194 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.568920 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.568878 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-551dd1\"/\"distinct-2-w3-551dd1-kserve-self-signed-certs\"" Oct 30 12:57:03.572460 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.572425 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl"] Oct 30 12:57:03.589479 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.589445 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tls-certs\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.589634 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.589492 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhd4x\" (UniqueName: \"kubernetes.io/projected/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-kube-api-access-rhd4x\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.589634 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.589582 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-model-cache\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.589634 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.589613 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tmp-dir\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.589808 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.589696 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-dshm\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.589808 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.589724 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-home\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.677081 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.677047 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:03.690361 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.690337 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tls-certs\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.690483 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.690375 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhd4x\" (UniqueName: \"kubernetes.io/projected/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-kube-api-access-rhd4x\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.690529 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.690493 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-model-cache\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.690567 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.690531 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tmp-dir\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.690621 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.690606 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-dshm\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.690680 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.690650 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-home\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.690939 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.690913 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-model-cache\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.691025 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.690937 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tmp-dir\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.691025 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.691016 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-home\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.692847 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.692816 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-dshm\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.693121 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.693103 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tls-certs\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.736535 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.736466 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhd4x\" (UniqueName: \"kubernetes.io/projected/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-kube-api-access-rhd4x\") pod \"distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:03.851624 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:03.851523 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:04.152228 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:04.152208 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl"] Oct 30 12:57:04.154550 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:57:04.154517 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9aa5bf2_ad4f_4bca_b301_8c9f1812b859.slice/crio-e027045c4a8b2aca7f135413388d4d2995b2e6aa224c92e8876d7c9ff8ff05e0 WatchSource:0}: Error finding container e027045c4a8b2aca7f135413388d4d2995b2e6aa224c92e8876d7c9ff8ff05e0: Status 404 returned error can't find the container with id e027045c4a8b2aca7f135413388d4d2995b2e6aa224c92e8876d7c9ff8ff05e0 Oct 30 12:57:04.588573 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:04.588490 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:05.084336 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:05.084295 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" event={"ID":"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859","Type":"ContainerStarted","Data":"67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5"} Oct 30 12:57:05.084336 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:05.084341 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" event={"ID":"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859","Type":"ContainerStarted","Data":"e027045c4a8b2aca7f135413388d4d2995b2e6aa224c92e8876d7c9ff8ff05e0"} Oct 30 12:57:05.127423 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:05.127371 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podStartSLOduration=2.127355782 podStartE2EDuration="2.127355782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:05.125570787 +0000 UTC m=+1443.252371695" watchObservedRunningTime="2026-10-30 12:57:05.127355782 +0000 UTC m=+1443.254156679" Oct 30 12:57:05.852100 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:05.852052 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:06.516314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:06.514379 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:08.457911 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:08.457880 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:08.706167 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:08.706128 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:09.052679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:09.052653 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:57:09.127244 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:09.127207 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4"] Oct 30 12:57:10.054282 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:10.054238 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:11.568112 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:11.568073 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:12.835291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:12.835198 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:13.074077 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:13.074031 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:13.852178 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:13.852145 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:13.865246 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:13.865223 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:14.128332 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.128230 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" podUID="bb3c06e3-5e09-4c97-95d7-f0672c4cc246" containerName="payload-processing" containerID="cri-o://d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a" gracePeriod=25 Oct 30 12:57:14.161072 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.161045 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:57:14.408360 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.408338 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:57:14.447313 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.447286 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-provider-credentials\") pod \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " Oct 30 12:57:14.447432 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.447335 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-plugins-config-volume\") pod \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " Oct 30 12:57:14.447505 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.447434 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-kube-api-access-6ngg8\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-kube-api-access-6ngg8\") pod \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " Oct 30 12:57:14.447566 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.447533 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-routing-overlay-volume\") pod \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\" (UID: \"bb3c06e3-5e09-4c97-95d7-f0672c4cc246\") " Oct 30 12:57:14.447804 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.447782 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-plugins-config-volume" pod "bb3c06e3-5e09-4c97-95d7-f0672c4cc246" (UID: "bb3c06e3-5e09-4c97-95d7-f0672c4cc246"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:57:14.447893 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.447870 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-routing-overlay-volume" pod "bb3c06e3-5e09-4c97-95d7-f0672c4cc246" (UID: "bb3c06e3-5e09-4c97-95d7-f0672c4cc246"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:57:14.448031 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.448018 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:14.448087 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.448033 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:14.449773 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.449746 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-kube-api-access-6ngg8" pod "bb3c06e3-5e09-4c97-95d7-f0672c4cc246" (UID: "bb3c06e3-5e09-4c97-95d7-f0672c4cc246"). InnerVolumeSpecName "kube-api-access-6ngg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:14.449970 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.449950 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-provider-credentials" pod "bb3c06e3-5e09-4c97-95d7-f0672c4cc246" (UID: "bb3c06e3-5e09-4c97-95d7-f0672c4cc246"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:14.549576 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.549548 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:14.549576 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.549575 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngg8\" (UniqueName: \"kubernetes.io/projected/bb3c06e3-5e09-4c97-95d7-f0672c4cc246-kube-api-access-6ngg8\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:14.690949 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.690714 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:14.903237 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:14.903200 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:15.156891 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:15.156858 5515 generic.go:356] "Generic (PLEG): container finished" podID="bb3c06e3-5e09-4c97-95d7-f0672c4cc246" containerID="d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a" exitCode=0 Oct 30 12:57:15.157078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:15.156895 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" Oct 30 12:57:15.157078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:15.156949 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" event={"ID":"bb3c06e3-5e09-4c97-95d7-f0672c4cc246","Type":"ContainerDied","Data":"d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a"} Oct 30 12:57:15.157078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:15.156998 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4" event={"ID":"bb3c06e3-5e09-4c97-95d7-f0672c4cc246","Type":"ContainerDied","Data":"3f7841caa94f743645d1e981bb325e5e31a0e2f39fde51ed281eb7c1558a2b80"} Oct 30 12:57:15.157078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:15.157026 5515 scope.go:122] "RemoveContainer" containerID="d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a" Oct 30 12:57:15.167602 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:15.167587 5515 scope.go:122] "RemoveContainer" containerID="d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a" Oct 30 12:57:15.167897 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:57:15.167872 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a\": container with ID starting with d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a not found: ID does not exist" containerID="d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a" Oct 30 12:57:15.167962 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:15.167910 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a"} err="failed to get container status \"d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a\": rpc error: code = NotFound desc = could not find container \"d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a\": container with ID starting with d5fa6ce65f4a4088563cfe8a53f1b20ef6110fe1abc3ef575b0d48dc9505a58a not found: ID does not exist" Oct 30 12:57:15.207942 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:15.207919 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4"] Oct 30 12:57:15.219711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:15.219677 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-psdq4"] Oct 30 12:57:16.000788 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:16.000756 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3c06e3-5e09-4c97-95d7-f0672c4cc246" path="/var/lib/kubelet/pods/bb3c06e3-5e09-4c97-95d7-f0672c4cc246/volumes" Oct 30 12:57:16.376880 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:16.376799 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:17.984873 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:17.984838 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:19.099730 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:19.099697 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:57:19.521477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:19.521445 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:22.556382 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:22.556328 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:23.691239 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.691203 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d"] Oct 30 12:57:23.691722 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.691708 5515 cpu_manager.go:414] "containerMap: removing container" podUID="bb3c06e3-5e09-4c97-95d7-f0672c4cc246" containerName="payload-processing" Oct 30 12:57:23.691765 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.691723 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3c06e3-5e09-4c97-95d7-f0672c4cc246" containerName="payload-processing" Oct 30 12:57:23.691842 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.691833 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3c06e3-5e09-4c97-95d7-f0672c4cc246" containerName="payload-processing" Oct 30 12:57:23.695024 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.695010 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.722067 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.722037 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d"] Oct 30 12:57:23.835871 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.835837 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.835871 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.835873 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.836102 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.835956 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.836102 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.835987 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.836102 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.836022 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe68e-ab48-41a6-bcdb-ab612596352b-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.836226 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.836134 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.836226 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.836187 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.836335 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.836251 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p7r2\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-kube-api-access-8p7r2\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.836374 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.836333 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.937624 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.937590 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.937758 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.937699 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p7r2\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-kube-api-access-8p7r2\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.937758 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.937741 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.937879 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.937799 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.937879 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.937824 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.937879 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.937867 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.938022 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.937907 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.938022 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.937964 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe68e-ab48-41a6-bcdb-ab612596352b-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.938022 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.938010 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.938309 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.938257 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.938478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.938328 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.938560 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.938362 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.938560 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.938395 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.938694 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.938665 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe68e-ab48-41a6-bcdb-ab612596352b-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.940217 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.940193 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.940397 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.940378 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.961586 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.961529 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:23.964764 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:23.964741 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p7r2\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-kube-api-access-8p7r2\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:24.009277 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:24.009247 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:24.335821 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:24.335796 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d"] Oct 30 12:57:24.337409 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:57:24.337383 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0abe68e_ab48_41a6_bcdb_ab612596352b.slice/crio-d00340fdbe9fbec55ae47cbf1f06dc6d04e068f957b5c46b09f69f671d499d5f WatchSource:0}: Error finding container d00340fdbe9fbec55ae47cbf1f06dc6d04e068f957b5c46b09f69f671d499d5f: Status 404 returned error can't find the container with id d00340fdbe9fbec55ae47cbf1f06dc6d04e068f957b5c46b09f69f671d499d5f Oct 30 12:57:24.339775 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:24.339745 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:57:24.339850 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:24.339811 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:57:24.339850 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:24.339838 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:57:24.520437 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:24.517672 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:24.727180 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:24.727106 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:25.332103 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:25.332052 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" event={"ID":"b0abe68e-ab48-41a6-bcdb-ab612596352b","Type":"ContainerStarted","Data":"662bbfeb975500dd20c78f781ba94f1371d4f24bd3621192523d81bffa2c306d"} Oct 30 12:57:25.332362 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:25.332160 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" event={"ID":"b0abe68e-ab48-41a6-bcdb-ab612596352b","Type":"ContainerStarted","Data":"d00340fdbe9fbec55ae47cbf1f06dc6d04e068f957b5c46b09f69f671d499d5f"} Oct 30 12:57:25.401298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:25.394758 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" podStartSLOduration=2.394741644 podStartE2EDuration="2.394741644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:25.392197768 +0000 UTC m=+1463.518998667" watchObservedRunningTime="2026-10-30 12:57:25.394741644 +0000 UTC m=+1463.521542543" Oct 30 12:57:26.012292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:26.011483 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:57:26.012292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:26.011548 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 12:57:26.026085 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:26.026056 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:26.467290 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:26.467229 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:27.012301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:27.011632 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:57:27.012301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:27.011693 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 12:57:28.018719 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:28.018687 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:57:28.019165 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:28.018743 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 12:57:28.163321 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:28.159332 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:28.990941 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:57:28.990902 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:57:29.018318 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:29.014684 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:57:29.018318 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:29.014746 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 12:57:29.691911 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:29.691874 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:30.016868 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:30.016366 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:57:30.016868 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:30.016429 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 12:57:31.010189 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:31.010152 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:57:31.010573 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:31.010219 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 12:57:31.289620 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:31.289526 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:32.014521 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:32.014496 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:32.014926 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:32.014765 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:32.015390 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:32.015375 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 12:57:32.989516 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:32.989485 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:34.565608 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:34.565557 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:36.174013 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:36.173972 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:38.193067 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:38.193031 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:39.201172 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.201137 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:39.407514 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.407481 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:39.548655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.548585 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-kw469"] Oct 30 12:57:39.553539 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.553522 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.586084 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.586046 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-kw469"] Oct 30 12:57:39.638754 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.638723 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-routing-overlay-volume\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.638873 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.638763 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-plugins-config-volume\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.638914 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.638881 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsf2x\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-kube-api-access-wsf2x\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.638949 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.638923 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-provider-credentials\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.739726 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.739686 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsf2x\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-kube-api-access-wsf2x\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.739726 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.739729 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-provider-credentials\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.739927 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.739841 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-routing-overlay-volume\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.739927 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.739876 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-plugins-config-volume\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.740520 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.740496 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-routing-overlay-volume\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.740736 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.740719 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-plugins-config-volume\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.742313 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.742284 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-provider-credentials\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.762829 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.762808 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsf2x\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-kube-api-access-wsf2x\") pod \"payload-processing-external-model-cdc946f8b-kw469\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:39.867750 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:39.867661 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:40.031083 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:40.031036 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:40.174361 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:40.174331 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-kw469"] Oct 30 12:57:40.174496 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:57:40.174342 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6b54e2_53df_4338_ab0b_1d99e7488540.slice/crio-7cc7a729f14f25d5fc3f6d848624701ac56fe1e43d7ff3401dab3b1456130a79 WatchSource:0}: Error finding container 7cc7a729f14f25d5fc3f6d848624701ac56fe1e43d7ff3401dab3b1456130a79: Status 404 returned error can't find the container with id 7cc7a729f14f25d5fc3f6d848624701ac56fe1e43d7ff3401dab3b1456130a79 Oct 30 12:57:40.419369 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:40.419332 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" event={"ID":"1c6b54e2-53df-4338-ab0b-1d99e7488540","Type":"ContainerStarted","Data":"57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4"} Oct 30 12:57:40.419782 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:40.419413 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" event={"ID":"1c6b54e2-53df-4338-ab0b-1d99e7488540","Type":"ContainerStarted","Data":"7cc7a729f14f25d5fc3f6d848624701ac56fe1e43d7ff3401dab3b1456130a79"} Oct 30 12:57:40.419782 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:40.419475 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:40.462624 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:40.462544 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" podStartSLOduration=1.462530936 podStartE2EDuration="1.462530936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:40.460761203 +0000 UTC m=+1478.587562102" watchObservedRunningTime="2026-10-30 12:57:40.462530936 +0000 UTC m=+1478.589331872" Oct 30 12:57:41.555467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:41.555420 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:41.873883 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:41.873799 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:43.598947 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:43.598902 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:45.273715 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:45.273674 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:45.745137 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:45.745104 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:57:46.946078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:46.946038 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:48.587143 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:48.587092 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:50.197198 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:50.197162 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:51.428732 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:51.428705 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:51.532828 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:51.532793 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-947b6b675-9k2tf"] Oct 30 12:57:51.762539 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:51.762455 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-kw469"] Oct 30 12:57:51.847768 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:51.847724 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:52.068580 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:52.068500 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:52.774903 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:52.774867 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:53.683607 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:53.683572 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:55.130115 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:55.130079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:56.533875 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.533823 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" podUID="5f03984b-2100-4f15-821e-e7a15a7a41ca" containerName="payload-processing" containerID="cri-o://98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163" gracePeriod=25 Oct 30 12:57:56.669464 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.669429 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:56.763229 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.763182 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" podUID="1c6b54e2-53df-4338-ab0b-1d99e7488540" containerName="payload-processing" containerID="cri-o://57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4" gracePeriod=25 Oct 30 12:57:56.892052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.892028 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:57:56.982301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.982236 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-provider-credentials\") pod \"5f03984b-2100-4f15-821e-e7a15a7a41ca\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " Oct 30 12:57:56.982467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.982351 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-routing-overlay-volume\") pod \"5f03984b-2100-4f15-821e-e7a15a7a41ca\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " Oct 30 12:57:56.982467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.982412 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-kube-api-access-hbkct\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-kube-api-access-hbkct\") pod \"5f03984b-2100-4f15-821e-e7a15a7a41ca\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " Oct 30 12:57:56.982467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.982452 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-plugins-config-volume\") pod \"5f03984b-2100-4f15-821e-e7a15a7a41ca\" (UID: \"5f03984b-2100-4f15-821e-e7a15a7a41ca\") " Oct 30 12:57:56.983014 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.982762 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-routing-overlay-volume" pod "5f03984b-2100-4f15-821e-e7a15a7a41ca" (UID: "5f03984b-2100-4f15-821e-e7a15a7a41ca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:57:56.983014 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.982904 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:56.983437 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.983410 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-plugins-config-volume" pod "5f03984b-2100-4f15-821e-e7a15a7a41ca" (UID: "5f03984b-2100-4f15-821e-e7a15a7a41ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:57:56.985293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.985150 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-provider-credentials" pod "5f03984b-2100-4f15-821e-e7a15a7a41ca" (UID: "5f03984b-2100-4f15-821e-e7a15a7a41ca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:56.985293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:56.985238 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-kube-api-access-hbkct" pod "5f03984b-2100-4f15-821e-e7a15a7a41ca" (UID: "5f03984b-2100-4f15-821e-e7a15a7a41ca"). InnerVolumeSpecName "kube-api-access-hbkct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:57.028722 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.028700 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:57.083961 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.083891 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-routing-overlay-volume\") pod \"1c6b54e2-53df-4338-ab0b-1d99e7488540\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " Oct 30 12:57:57.083961 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.083942 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-plugins-config-volume\") pod \"1c6b54e2-53df-4338-ab0b-1d99e7488540\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " Oct 30 12:57:57.084181 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.084053 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-provider-credentials\") pod \"1c6b54e2-53df-4338-ab0b-1d99e7488540\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " Oct 30 12:57:57.084181 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.084133 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-kube-api-access-wsf2x\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-kube-api-access-wsf2x\") pod \"1c6b54e2-53df-4338-ab0b-1d99e7488540\" (UID: \"1c6b54e2-53df-4338-ab0b-1d99e7488540\") " Oct 30 12:57:57.084361 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.084336 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-routing-overlay-volume" pod "1c6b54e2-53df-4338-ab0b-1d99e7488540" (UID: "1c6b54e2-53df-4338-ab0b-1d99e7488540"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:57:57.084606 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.084569 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-plugins-config-volume" pod "1c6b54e2-53df-4338-ab0b-1d99e7488540" (UID: "1c6b54e2-53df-4338-ab0b-1d99e7488540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:57:57.089287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.085183 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f03984b-2100-4f15-821e-e7a15a7a41ca-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:57.089287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.086328 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:57.089287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.086348 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:57.089287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.086361 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbkct\" (UniqueName: \"kubernetes.io/projected/5f03984b-2100-4f15-821e-e7a15a7a41ca-kube-api-access-hbkct\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:57.089287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.087423 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-provider-credentials" pod "1c6b54e2-53df-4338-ab0b-1d99e7488540" (UID: "1c6b54e2-53df-4338-ab0b-1d99e7488540"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:57.089287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.088925 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-kube-api-access-wsf2x" pod "1c6b54e2-53df-4338-ab0b-1d99e7488540" (UID: "1c6b54e2-53df-4338-ab0b-1d99e7488540"). InnerVolumeSpecName "kube-api-access-wsf2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:57:57.187291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.187246 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:57.187291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.187292 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsf2x\" (UniqueName: \"kubernetes.io/projected/1c6b54e2-53df-4338-ab0b-1d99e7488540-kube-api-access-wsf2x\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:57.187451 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.187302 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6b54e2-53df-4338-ab0b-1d99e7488540-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 12:57:57.516657 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.516625 5515 generic.go:356] "Generic (PLEG): container finished" podID="5f03984b-2100-4f15-821e-e7a15a7a41ca" containerID="98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163" exitCode=0 Oct 30 12:57:57.516852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.516669 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" Oct 30 12:57:57.516852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.516712 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" event={"ID":"5f03984b-2100-4f15-821e-e7a15a7a41ca","Type":"ContainerDied","Data":"98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163"} Oct 30 12:57:57.516852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.516765 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-947b6b675-9k2tf" event={"ID":"5f03984b-2100-4f15-821e-e7a15a7a41ca","Type":"ContainerDied","Data":"53ddb709b4ac2a8dc7f589e3961fdf7476f920be5a20b1ee1b3b5e430bc9792d"} Oct 30 12:57:57.516852 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.516792 5515 scope.go:122] "RemoveContainer" containerID="98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163" Oct 30 12:57:57.518158 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.518129 5515 generic.go:356] "Generic (PLEG): container finished" podID="1c6b54e2-53df-4338-ab0b-1d99e7488540" containerID="57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4" exitCode=0 Oct 30 12:57:57.518305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.518158 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" event={"ID":"1c6b54e2-53df-4338-ab0b-1d99e7488540","Type":"ContainerDied","Data":"57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4"} Oct 30 12:57:57.518305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.518184 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" Oct 30 12:57:57.518305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.518198 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc946f8b-kw469" event={"ID":"1c6b54e2-53df-4338-ab0b-1d99e7488540","Type":"ContainerDied","Data":"7cc7a729f14f25d5fc3f6d848624701ac56fe1e43d7ff3401dab3b1456130a79"} Oct 30 12:57:57.532060 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.532043 5515 scope.go:122] "RemoveContainer" containerID="98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163" Oct 30 12:57:57.532398 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:57:57.532375 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163\": container with ID starting with 98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163 not found: ID does not exist" containerID="98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163" Oct 30 12:57:57.532480 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.532405 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163"} err="failed to get container status \"98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163\": rpc error: code = NotFound desc = could not find container \"98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163\": container with ID starting with 98bbe56d2ab944097dafdfd46aae4036b568c39e28ad9536d57c7bb216cac163 not found: ID does not exist" Oct 30 12:57:57.532480 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.532423 5515 scope.go:122] "RemoveContainer" containerID="57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4" Oct 30 12:57:57.541868 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.541685 5515 scope.go:122] "RemoveContainer" containerID="57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4" Oct 30 12:57:57.542089 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:57:57.541929 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4\": container with ID starting with 57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4 not found: ID does not exist" containerID="57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4" Oct 30 12:57:57.542089 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.541955 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4"} err="failed to get container status \"57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4\": rpc error: code = NotFound desc = could not find container \"57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4\": container with ID starting with 57efc04f59839d1f39861a40222627c9581de22460f876f077d9526ff70f79e4 not found: ID does not exist" Oct 30 12:57:57.568193 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.568172 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-kw469"] Oct 30 12:57:57.579982 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.579962 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc946f8b-kw469"] Oct 30 12:57:57.613999 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.613973 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-947b6b675-9k2tf"] Oct 30 12:57:57.616164 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:57.616143 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-947b6b675-9k2tf"] Oct 30 12:57:58.000856 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:58.000824 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6b54e2-53df-4338-ab0b-1d99e7488540" path="/var/lib/kubelet/pods/1c6b54e2-53df-4338-ab0b-1d99e7488540/volumes" Oct 30 12:57:58.001196 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:58.001183 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f03984b-2100-4f15-821e-e7a15a7a41ca" path="/var/lib/kubelet/pods/5f03984b-2100-4f15-821e-e7a15a7a41ca/volumes" Oct 30 12:57:58.432494 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:57:58.432458 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:59.048291 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:57:59.047537 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:58:00.893153 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:00.882358 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:58:00.893153 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:00.882416 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:58:00.893153 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:00.884378 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7cq2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:58:00.893153 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:00.884423 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:58:01.979679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:01.979642 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:03.654655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:03.654588 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:05.375477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:05.375440 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:06.978287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:06.978219 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:07.375755 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.375675 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql"] Oct 30 12:58:07.376230 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.376213 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5f03984b-2100-4f15-821e-e7a15a7a41ca" containerName="payload-processing" Oct 30 12:58:07.376307 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.376231 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f03984b-2100-4f15-821e-e7a15a7a41ca" containerName="payload-processing" Oct 30 12:58:07.376307 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.376255 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1c6b54e2-53df-4338-ab0b-1d99e7488540" containerName="payload-processing" Oct 30 12:58:07.376307 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.376263 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6b54e2-53df-4338-ab0b-1d99e7488540" containerName="payload-processing" Oct 30 12:58:07.376461 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.376446 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f03984b-2100-4f15-821e-e7a15a7a41ca" containerName="payload-processing" Oct 30 12:58:07.376584 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.376466 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6b54e2-53df-4338-ab0b-1d99e7488540" containerName="payload-processing" Oct 30 12:58:07.381783 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.381764 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.407387 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.407345 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql"] Oct 30 12:58:07.486472 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.486435 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.486472 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.486477 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.486694 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.486544 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.486694 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.486627 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.486694 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.486680 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vgn\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-kube-api-access-k2vgn\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.486797 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.486730 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.486797 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.486762 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5879660-fa09-42c2-ba35-6ef6a6c67930-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.486862 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.486845 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.486897 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.486878 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587488 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587455 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5879660-fa09-42c2-ba35-6ef6a6c67930-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587673 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587550 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587673 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587588 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587673 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587628 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587673 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587654 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587887 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587686 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587887 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587733 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587997 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587919 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vgn\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-kube-api-access-k2vgn\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587997 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587965 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.587997 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.587978 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.588161 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.588074 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.588211 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.588163 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.588211 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.588187 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.588322 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.588302 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5879660-fa09-42c2-ba35-6ef6a6c67930-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.590128 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.590104 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.590261 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.590243 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.610927 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.610902 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.611054 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.611036 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vgn\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-kube-api-access-k2vgn\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:07.695321 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:07.695293 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:08.010959 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:58:08.010928 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5879660_fa09_42c2_ba35_6ef6a6c67930.slice/crio-fd6e9d3975592fa19f7416f63a39ecc803e90149b36183806fb41ac953d82ab9 WatchSource:0}: Error finding container fd6e9d3975592fa19f7416f63a39ecc803e90149b36183806fb41ac953d82ab9: Status 404 returned error can't find the container with id fd6e9d3975592fa19f7416f63a39ecc803e90149b36183806fb41ac953d82ab9 Oct 30 12:58:08.012173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.012148 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql"] Oct 30 12:58:08.013198 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.013168 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:08.013251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.013231 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:08.013311 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.013259 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:08.645655 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.645618 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:08.656814 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.656784 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" event={"ID":"e5879660-fa09-42c2-ba35-6ef6a6c67930","Type":"ContainerStarted","Data":"c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a"} Oct 30 12:58:08.656992 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.656836 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" event={"ID":"e5879660-fa09-42c2-ba35-6ef6a6c67930","Type":"ContainerStarted","Data":"fd6e9d3975592fa19f7416f63a39ecc803e90149b36183806fb41ac953d82ab9"} Oct 30 12:58:08.702290 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.697362 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:08.713292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.709121 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" podStartSLOduration=1.709103411 podStartE2EDuration="1.709103411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:08.706901889 +0000 UTC m=+1506.833702787" watchObservedRunningTime="2026-10-30 12:58:08.709103411 +0000 UTC m=+1506.835904310" Oct 30 12:58:08.850258 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:08.850228 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:09.722729 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.722694 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:58:09.723317 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.723281 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 12:58:09.727802 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.727776 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv"] Oct 30 12:58:09.733184 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.733104 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:09.749794 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.749768 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv"] Oct 30 12:58:09.759184 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.759160 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bb86fd\"/\"distinct-w2-bb86fd-kserve-self-signed-certs\"" Oct 30 12:58:09.882473 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.882439 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:09.930030 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.929969 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-home\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:09.930362 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.930335 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tls-certs\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:09.930952 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.930512 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbzp\" (UniqueName: \"kubernetes.io/projected/91f6d43e-ecc0-47cb-890e-c1958dc967f2-kube-api-access-dxbzp\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:09.930952 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.930615 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-dshm\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:09.930952 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.930737 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tmp-dir\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:09.930952 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:09.930774 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-model-cache\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.035116 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.035018 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tmp-dir\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.035410 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.035386 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-model-cache\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.035593 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.035577 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-home\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.035744 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.035730 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tls-certs\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.035864 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.035848 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbzp\" (UniqueName: \"kubernetes.io/projected/91f6d43e-ecc0-47cb-890e-c1958dc967f2-kube-api-access-dxbzp\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.036023 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.036007 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-dshm\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.036862 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.036688 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-home\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.037684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.036686 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tmp-dir\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.037977 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.037892 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-model-cache\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.039348 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.039003 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-dshm\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.040817 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.040790 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tls-certs\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.063667 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.063531 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbzp\" (UniqueName: \"kubernetes.io/projected/91f6d43e-ecc0-47cb-890e-c1958dc967f2-kube-api-access-dxbzp\") pod \"distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.110295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.107588 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:10.346895 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.346803 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:10.671111 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.671086 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv"] Oct 30 12:58:10.671961 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:58:10.671937 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91f6d43e_ecc0_47cb_890e_c1958dc967f2.slice/crio-09ad46c91b6ebf13c1e74ffde58c56c650a072788a33cb8d84667fbad55d9fc8 WatchSource:0}: Error finding container 09ad46c91b6ebf13c1e74ffde58c56c650a072788a33cb8d84667fbad55d9fc8: Status 404 returned error can't find the container with id 09ad46c91b6ebf13c1e74ffde58c56c650a072788a33cb8d84667fbad55d9fc8 Oct 30 12:58:10.718374 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.717735 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:58:10.718374 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:10.717789 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 12:58:11.676442 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:11.676411 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" event={"ID":"91f6d43e-ecc0-47cb-890e-c1958dc967f2","Type":"ContainerStarted","Data":"9a7a6ed7152867ac1b94d34471c75c51ca32f8bee599023a8d9f897b4c2bee8e"} Oct 30 12:58:11.676911 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:11.676453 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" event={"ID":"91f6d43e-ecc0-47cb-890e-c1958dc967f2","Type":"ContainerStarted","Data":"09ad46c91b6ebf13c1e74ffde58c56c650a072788a33cb8d84667fbad55d9fc8"} Oct 30 12:58:11.699296 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:11.697879 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:58:11.699296 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:11.697936 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 12:58:11.727563 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:11.727500 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podStartSLOduration=2.727485857 podStartE2EDuration="2.727485857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:11.726340981 +0000 UTC m=+1509.853141878" watchObservedRunningTime="2026-10-30 12:58:11.727485857 +0000 UTC m=+1509.854286756" Oct 30 12:58:11.787602 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:11.787566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:12.347087 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:12.347056 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:12.712292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:12.705621 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:58:12.712292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:12.705679 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 12:58:13.577251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:13.577202 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:13.698659 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:13.698624 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:58:13.698911 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:13.698882 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 12:58:14.696021 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:14.695989 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:58:14.696537 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:14.696053 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 12:58:15.101346 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:15.099836 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:15.694744 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:15.694709 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:58:15.699757 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:15.699734 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:15.700136 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:15.700090 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:15.700982 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:15.700964 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 12:58:16.721965 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:16.721922 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:18.668735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:18.668699 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:20.163053 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:20.163019 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:20.347592 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:20.347563 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:20.360695 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:20.360672 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:20.377554 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:20.377529 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:20.781233 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:20.781205 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:58:21.139282 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.139249 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj"] Oct 30 12:58:21.143178 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.143161 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.173674 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.173639 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6ff0e6-openshift-default-dockercfg-xpjcg\"" Oct 30 12:58:21.177660 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.177641 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj"] Oct 30 12:58:21.204968 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.204938 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1018d8-6578-406e-a683-3c183fb963d0-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.205081 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.204979 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.205081 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.205013 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.205166 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.205111 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjrr\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-kube-api-access-kfjrr\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.205166 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.205152 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.205298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.205259 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.205345 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.205332 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.205376 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.205355 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1018d8-6578-406e-a683-3c183fb963d0-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.205423 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.205386 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306389 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306360 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306526 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306421 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306526 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306446 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1018d8-6578-406e-a683-3c183fb963d0-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306526 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306478 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306526 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306505 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1018d8-6578-406e-a683-3c183fb963d0-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306741 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306538 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306741 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306584 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306741 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306611 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjrr\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-kube-api-access-kfjrr\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306741 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306648 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306940 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306893 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.306998 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.306937 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.307053 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.307005 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.307126 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.307104 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.307305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.307254 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1018d8-6578-406e-a683-3c183fb963d0-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.308813 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.308791 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.309025 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.309009 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1018d8-6578-406e-a683-3c183fb963d0-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.337017 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.336994 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.339952 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.339934 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjrr\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-kube-api-access-kfjrr\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.348555 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.348529 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd"] Oct 30 12:58:21.354294 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.354249 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.408960 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.408925 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd"] Oct 30 12:58:21.457765 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.457734 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:21.511313 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.511246 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.511515 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.511490 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.511587 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.511564 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.511659 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.511645 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.511729 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.511710 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.511781 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.511765 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44trj\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-kube-api-access-44trj\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.511825 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.511803 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.511971 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.511948 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe03a9a4-eae9-418e-b91c-73a175fc4804-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.512035 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.512015 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613356 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613321 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe03a9a4-eae9-418e-b91c-73a175fc4804-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613520 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613368 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613520 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613479 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613613 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613583 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613613 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613608 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613714 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613643 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613714 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613672 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613820 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613717 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44trj\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-kube-api-access-44trj\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613820 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613745 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613930 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613823 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.613930 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613753 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.614030 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.613981 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.614159 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.614135 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe03a9a4-eae9-418e-b91c-73a175fc4804-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.614263 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.614245 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.615805 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.615780 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.616045 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.616029 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.638257 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.638231 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.638508 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.638491 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44trj\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-kube-api-access-44trj\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.728071 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.728007 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:21.794501 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.794471 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj"] Oct 30 12:58:21.798575 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:58:21.798536 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1018d8_6578_406e_a683_3c183fb963d0.slice/crio-f5aa2604df59bd368859787513fa991787171634e946a1f59bfd1a938294ae0f WatchSource:0}: Error finding container f5aa2604df59bd368859787513fa991787171634e946a1f59bfd1a938294ae0f: Status 404 returned error can't find the container with id f5aa2604df59bd368859787513fa991787171634e946a1f59bfd1a938294ae0f Oct 30 12:58:21.802383 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.802350 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:21.802467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.802414 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:21.802467 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.802442 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:21.806962 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:21.806934 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:22.102337 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.102310 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd"] Oct 30 12:58:22.103250 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:58:22.103224 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe03a9a4_eae9_418e_b91c_73a175fc4804.slice/crio-f5862a328f0a1a169f1f05cd71ef858048f6a01f8c2870a7ed13a3f953b9672b WatchSource:0}: Error finding container f5862a328f0a1a169f1f05cd71ef858048f6a01f8c2870a7ed13a3f953b9672b: Status 404 returned error can't find the container with id f5862a328f0a1a169f1f05cd71ef858048f6a01f8c2870a7ed13a3f953b9672b Oct 30 12:58:22.105735 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.105702 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:22.105837 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.105766 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:22.105837 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.105798 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:22.446700 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.446672 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b"] Oct 30 12:58:22.450900 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.450875 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.477313 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.476335 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b"] Oct 30 12:58:22.533004 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.532928 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vts\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-kube-api-access-47vts\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.533200 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.533095 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.533288 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.533204 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.533443 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.533402 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.533521 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.533459 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64555a8-b762-489a-b00b-e952c6253a85-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.533582 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.533569 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.533637 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.533607 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.533693 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.533652 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.533762 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.533740 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64555a8-b762-489a-b00b-e952c6253a85-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.634728 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.634693 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.634928 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.634839 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.634928 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.634882 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64555a8-b762-489a-b00b-e952c6253a85-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.634928 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.634920 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.635106 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.634955 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.635106 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.635003 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.635106 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.635036 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64555a8-b762-489a-b00b-e952c6253a85-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.635106 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.635095 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vts\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-kube-api-access-47vts\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.635322 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.635137 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.635322 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.635170 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.635436 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.635364 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.635523 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.635503 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.636102 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.636071 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.636597 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.636566 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64555a8-b762-489a-b00b-e952c6253a85-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.637726 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.637703 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.637888 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.637869 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64555a8-b762-489a-b00b-e952c6253a85-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.660804 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.660757 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.660980 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.660945 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vts\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-kube-api-access-47vts\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.769795 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.769714 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:22.850371 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.850318 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" event={"ID":"9d1018d8-6578-406e-a683-3c183fb963d0","Type":"ContainerStarted","Data":"cf3b0d9f8eb58d5b26abffe6ca3b15d0bd9bd8518499b8d9b176de43170d1eaa"} Oct 30 12:58:22.850578 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.850402 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" event={"ID":"9d1018d8-6578-406e-a683-3c183fb963d0","Type":"ContainerStarted","Data":"f5aa2604df59bd368859787513fa991787171634e946a1f59bfd1a938294ae0f"} Oct 30 12:58:22.852379 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.852229 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" event={"ID":"fe03a9a4-eae9-418e-b91c-73a175fc4804","Type":"ContainerStarted","Data":"d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5"} Oct 30 12:58:22.852379 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.852311 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" event={"ID":"fe03a9a4-eae9-418e-b91c-73a175fc4804","Type":"ContainerStarted","Data":"f5862a328f0a1a169f1f05cd71ef858048f6a01f8c2870a7ed13a3f953b9672b"} Oct 30 12:58:22.976557 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:22.976494 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podStartSLOduration=1.976473994 podStartE2EDuration="1.976473994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:22.928617299 +0000 UTC m=+1521.055418201" watchObservedRunningTime="2026-10-30 12:58:22.976473994 +0000 UTC m=+1521.103274893" Oct 30 12:58:23.147341 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.138621 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" podStartSLOduration=2.13860113 podStartE2EDuration="2.13860113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:22.975420299 +0000 UTC m=+1521.102221196" watchObservedRunningTime="2026-10-30 12:58:23.13860113 +0000 UTC m=+1521.265402028" Oct 30 12:58:23.147341 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.143174 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b"] Oct 30 12:58:23.152488 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:58:23.152452 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64555a8_b762_489a_b00b_e952c6253a85.slice/crio-1976206a2317d51a216154acb3b6ccf1ded12684236b6ab5e03eb12b9936aa50 WatchSource:0}: Error finding container 1976206a2317d51a216154acb3b6ccf1ded12684236b6ab5e03eb12b9936aa50: Status 404 returned error can't find the container with id 1976206a2317d51a216154acb3b6ccf1ded12684236b6ab5e03eb12b9936aa50 Oct 30 12:58:23.154820 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.154787 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:23.154882 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.154854 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:23.154925 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.154882 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:23.458342 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.458139 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:23.460912 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.460885 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:58:23.461107 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.461086 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 12:58:23.655802 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.652130 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:23.730124 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.730039 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:23.730447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.730423 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:58:23.730635 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.730609 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:58:23.868313 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.868256 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" event={"ID":"c64555a8-b762-489a-b00b-e952c6253a85","Type":"ContainerStarted","Data":"1b51ab7a0ceba1e0f08ec20bd311d49760e16c415881c20a7d388b408b4a9107"} Oct 30 12:58:23.868630 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.868613 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" event={"ID":"c64555a8-b762-489a-b00b-e952c6253a85","Type":"ContainerStarted","Data":"1976206a2317d51a216154acb3b6ccf1ded12684236b6ab5e03eb12b9936aa50"} Oct 30 12:58:23.884463 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.884432 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z"] Oct 30 12:58:23.889640 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.889457 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.968386 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z"] Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.969493 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.969541 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.969599 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.969623 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.969690 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.969793 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.969866 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.969947 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6t4g\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-kube-api-access-h6t4g\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.970293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.970006 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:23.971173 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:23.971125 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podStartSLOduration=1.9711104910000001 podStartE2EDuration="1.971110491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:23.950249961 +0000 UTC m=+1522.077050861" watchObservedRunningTime="2026-10-30 12:58:23.971110491 +0000 UTC m=+1522.097911390" Oct 30 12:58:24.078378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.077747 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.078378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.077796 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.078378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.077856 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.078378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.077912 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.078378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.077959 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.078378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.078009 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6t4g\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-kube-api-access-h6t4g\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.078378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.078050 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.078378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.078132 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.078378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.078169 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.079001 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.078773 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.079751 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.079720 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.080015 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.079993 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.080627 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.080608 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.080720 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.080665 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.081576 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.081553 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.082052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.082036 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.141725 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.131088 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6t4g\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-kube-api-access-h6t4g\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.156980 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.151810 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.282028 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.281990 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:24.462745 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.459773 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:58:24.462745 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.459827 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 12:58:24.742375 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.736752 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:58:24.742375 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.736812 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:58:24.759306 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.758039 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z"] Oct 30 12:58:24.776314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.775318 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:24.776647 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.776619 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:58:24.776807 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.776782 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:58:24.780196 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:58:24.780105 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df1bf5b_456f_4aea_bd62_c11eaa4ad45c.slice/crio-1161cf3e9e14cc86620f0984ee04ce8bedd1a7d0ff273dea1d1fd4698bc2df2e WatchSource:0}: Error finding container 1161cf3e9e14cc86620f0984ee04ce8bedd1a7d0ff273dea1d1fd4698bc2df2e: Status 404 returned error can't find the container with id 1161cf3e9e14cc86620f0984ee04ce8bedd1a7d0ff273dea1d1fd4698bc2df2e Oct 30 12:58:24.783359 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.783327 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:24.783462 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.783412 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:24.783530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.783458 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:24.890564 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:24.890532 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" event={"ID":"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c","Type":"ContainerStarted","Data":"1161cf3e9e14cc86620f0984ee04ce8bedd1a7d0ff273dea1d1fd4698bc2df2e"} Oct 30 12:58:25.459078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:25.458644 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:58:25.459078 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:25.458706 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 12:58:25.663219 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:25.663177 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:25.733178 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:25.732693 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:58:25.733178 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:25.732751 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:58:25.800292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:25.784327 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:58:25.800292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:25.784385 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:58:25.929291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:25.929137 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" event={"ID":"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c","Type":"ContainerStarted","Data":"902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db"} Oct 30 12:58:26.293324 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:26.284966 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:26.293324 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:26.285134 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 12:58:26.293324 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:26.285179 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 12:58:26.465304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:26.463062 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:58:26.465304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:26.463125 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 12:58:26.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:26.729778 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:58:26.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:26.729857 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:58:26.780292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:26.779596 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:58:26.780292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:26.779652 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:58:27.286304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:27.283647 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 12:58:27.286304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:27.283708 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 12:58:27.695293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:27.686157 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:58:27.695293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:27.686212 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 12:58:27.753293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:27.752741 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:58:27.753293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:27.752804 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:58:27.773292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:27.770942 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:58:27.773292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:27.770999 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:58:27.876299 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:27.872817 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 12:58:28.288295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:28.287679 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 12:58:28.288295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:28.287741 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 12:58:28.478296 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:28.476612 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:58:28.478296 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:28.476672 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 12:58:28.500320 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:28.491048 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:28.734292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:28.734135 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:58:28.734292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:28.734194 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:58:28.770434 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:28.770391 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:58:28.770894 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:28.770871 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:58:28.989023 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:58:28.988940 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:58:29.282799 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.282706 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 12:58:29.283038 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.283008 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 12:58:29.772327 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.746162 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 12:58:29.772327 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.746221 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 12:58:29.772901 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.772814 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:58:29.772901 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.772868 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:58:29.773462 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.773015 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:29.773665 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.773644 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:29.776356 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.776117 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 12:58:29.852711 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:29.852653 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" podStartSLOduration=6.852631499 podStartE2EDuration="6.852631499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:25.992649321 +0000 UTC m=+1524.119450288" watchObservedRunningTime="2026-10-30 12:58:29.852631499 +0000 UTC m=+1527.979432398" Oct 30 12:58:30.283145 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:30.283108 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 12:58:30.283338 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:30.283174 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 12:58:30.735361 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:30.735330 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:30.736138 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:30.735928 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:30.737139 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:30.737111 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 12:58:30.772371 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:30.771193 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 12:58:30.773228 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:30.771253 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 12:58:31.033341 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:31.032724 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:31.282699 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:31.282668 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 12:58:31.282854 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:31.282718 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 12:58:31.783608 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:31.783581 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:31.784046 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:31.784028 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:31.784866 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:31.784846 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 12:58:32.293841 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:32.293806 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 12:58:32.294179 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:32.294154 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 12:58:33.488292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:33.487598 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:33.493291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:33.489050 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:33.517291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:33.517061 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 12:58:34.155292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:34.154398 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:58:34.757777 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:34.757738 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:34.792320 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:34.777353 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:58:36.527504 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.527469 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt"] Oct 30 12:58:36.532911 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.532889 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.552573 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.552545 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt"] Oct 30 12:58:36.560587 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.560555 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"default-dockercfg-xk2th\"" Oct 30 12:58:36.560941 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.560882 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"distinct-2-w1-6ff0e6-kserve-self-signed-certs\"" Oct 30 12:58:36.563145 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.563117 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"openshift-service-ca.crt\"" Oct 30 12:58:36.563236 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.563162 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"kube-root-ca.crt\"" Oct 30 12:58:36.597389 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.597358 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-home\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.597515 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.597472 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-tmp-dir\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.597689 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.597667 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-model-cache\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.597753 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.597703 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6x66\" (UniqueName: \"kubernetes.io/projected/7616402d-0c8a-47c1-9f03-fcc54361145b-kube-api-access-k6x66\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.597822 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.597807 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7616402d-0c8a-47c1-9f03-fcc54361145b-tls-certs\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.597880 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.597864 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-dshm\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.688279 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.688245 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:36.698899 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.698871 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-model-cache\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.698899 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.698902 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6x66\" (UniqueName: \"kubernetes.io/projected/7616402d-0c8a-47c1-9f03-fcc54361145b-kube-api-access-k6x66\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.699110 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.698943 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7616402d-0c8a-47c1-9f03-fcc54361145b-tls-certs\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.699110 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.698990 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-dshm\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.699110 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.699020 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-home\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.699110 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.699071 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-tmp-dir\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.699384 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.699367 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-model-cache\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.699441 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.699408 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-home\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.699538 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.699519 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-tmp-dir\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.701171 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.701149 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-dshm\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.701466 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.701450 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7616402d-0c8a-47c1-9f03-fcc54361145b-tls-certs\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.723799 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.723769 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6x66\" (UniqueName: \"kubernetes.io/projected/7616402d-0c8a-47c1-9f03-fcc54361145b-kube-api-access-k6x66\") pod \"distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:36.846395 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:36.846319 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:37.265028 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:37.265000 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt"] Oct 30 12:58:37.265992 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:58:37.265969 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7616402d_0c8a_47c1_9f03_fcc54361145b.slice/crio-5170e068b762b893cde84eb7127440d66877f3f03cd178d4dd55909f4b5dee0b WatchSource:0}: Error finding container 5170e068b762b893cde84eb7127440d66877f3f03cd178d4dd55909f4b5dee0b: Status 404 returned error can't find the container with id 5170e068b762b893cde84eb7127440d66877f3f03cd178d4dd55909f4b5dee0b Oct 30 12:58:37.398288 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:37.396986 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:37.431104 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:37.431066 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" event={"ID":"7616402d-0c8a-47c1-9f03-fcc54361145b","Type":"ContainerStarted","Data":"b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60"} Oct 30 12:58:37.431104 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:37.431116 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" event={"ID":"7616402d-0c8a-47c1-9f03-fcc54361145b","Type":"ContainerStarted","Data":"5170e068b762b893cde84eb7127440d66877f3f03cd178d4dd55909f4b5dee0b"} Oct 30 12:58:37.476099 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:37.476040 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podStartSLOduration=1.476027134 podStartE2EDuration="1.476027134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:37.475609746 +0000 UTC m=+1535.602410645" watchObservedRunningTime="2026-10-30 12:58:37.476027134 +0000 UTC m=+1535.602828031" Oct 30 12:58:37.847291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:37.847239 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:39.039016 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:39.038956 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:40.691081 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.691037 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x"] Oct 30 12:58:40.696825 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.696798 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.697696 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.697667 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:40.731681 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.731644 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x"] Oct 30 12:58:40.755478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.755448 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.755619 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.755484 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.755619 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.755574 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.755892 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.755867 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.755977 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.755918 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.755977 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.755958 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.756060 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.756035 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.756152 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.756137 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.756192 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.756159 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8kg9\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-kube-api-access-r8kg9\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.858636 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.858600 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.858778 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.858654 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.858778 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.858672 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.858778 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.858690 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.858778 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.858705 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kg9\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-kube-api-access-r8kg9\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.859010 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.858786 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.859010 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.858806 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.859010 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.858849 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.859010 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.858942 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.859216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.859119 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.859216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.859146 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.859406 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.859293 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.859406 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.859373 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.859617 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.859591 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.862194 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.862171 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.862418 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.862398 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.886732 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.886698 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:40.886861 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:40.886771 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kg9\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-kube-api-access-r8kg9\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:41.016325 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:41.016235 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:41.332066 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:41.332033 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x"] Oct 30 12:58:41.332528 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:58:41.332504 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f24ede_fb9a_4d3f_8a62_c8074270d5c9.slice/crio-714f39b81cc60a279153ed8d346224a4dd9c1d05ecb74431fc5c001d9a59dcfd WatchSource:0}: Error finding container 714f39b81cc60a279153ed8d346224a4dd9c1d05ecb74431fc5c001d9a59dcfd: Status 404 returned error can't find the container with id 714f39b81cc60a279153ed8d346224a4dd9c1d05ecb74431fc5c001d9a59dcfd Oct 30 12:58:41.334873 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:41.334833 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:41.335038 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:41.335016 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:41.335117 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:41.335066 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:41.459349 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:41.459305 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" event={"ID":"91f24ede-fb9a-4d3f-8a62-c8074270d5c9","Type":"ContainerStarted","Data":"ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037"} Oct 30 12:58:41.459516 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:41.459385 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" event={"ID":"91f24ede-fb9a-4d3f-8a62-c8074270d5c9","Type":"ContainerStarted","Data":"714f39b81cc60a279153ed8d346224a4dd9c1d05ecb74431fc5c001d9a59dcfd"} Oct 30 12:58:41.506006 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:41.505936 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podStartSLOduration=1.505916206 podStartE2EDuration="1.505916206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:41.503795873 +0000 UTC m=+1539.630596794" watchObservedRunningTime="2026-10-30 12:58:41.505916206 +0000 UTC m=+1539.632717105" Oct 30 12:58:42.016538 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:42.016456 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:42.018112 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:42.018087 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 12:58:42.018229 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:42.018133 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 12:58:42.316865 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:42.316783 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:43.016798 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:43.016761 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 12:58:43.017450 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:43.017415 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 12:58:44.019672 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:44.019636 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 12:58:44.020255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:44.019704 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 12:58:44.134320 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:44.134281 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:45.016684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:45.016648 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 12:58:45.016974 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:45.016948 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 12:58:45.888152 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:45.888014 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:46.019251 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:46.019205 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 12:58:46.019489 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:46.019297 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 12:58:46.043880 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:46.043845 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:58:46.849039 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:46.849002 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:46.869347 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:46.869318 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:47.018660 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:47.018625 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 12:58:47.019238 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:47.019206 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 12:58:47.588255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:47.588216 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:58:48.033716 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:48.033679 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 12:58:48.034350 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:48.034317 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 12:58:48.544303 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:48.535979 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:49.021747 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:49.021707 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 12:58:49.022055 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:49.022022 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 12:58:49.132991 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:49.132948 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:58:49.508434 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:49.508389 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:58:50.020696 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:50.020661 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:50.021097 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:50.021071 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:50.021823 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:50.021803 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 12:58:50.592601 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:50.592548 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:51.160878 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.160797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:51.433527 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.433452 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz"] Oct 30 12:58:51.437462 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.437442 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.470058 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.470027 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz"] Oct 30 12:58:51.535302 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.535260 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.535465 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.535334 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.535682 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.535614 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.535682 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.535667 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.535809 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.535695 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.535809 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.535720 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.535809 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.535745 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25132bf1-77b7-4acb-940e-d36e07a12a76-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.535809 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.535792 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.535956 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.535839 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrxc\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-kube-api-access-frrxc\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637227 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637195 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637239 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637260 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637359 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637392 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25132bf1-77b7-4acb-940e-d36e07a12a76-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637429 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637473 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrxc\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-kube-api-access-frrxc\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637567 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637645 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637629 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637972 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637664 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.637972 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.637960 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.638228 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.638206 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.638324 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.638242 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.638324 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.638304 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25132bf1-77b7-4acb-940e-d36e07a12a76-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.639700 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.639679 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.640069 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.640051 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.660587 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.660563 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.661559 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.661539 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrxc\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-kube-api-access-frrxc\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:51.752052 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:51.751984 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:52.069445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:52.069421 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz"] Oct 30 12:58:52.070098 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:58:52.070079 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25132bf1_77b7_4acb_940e_d36e07a12a76.slice/crio-9bd2408e8eb4a7e2d89ea23a388d9e3cbc5c632044f98564d26ce4f86bd50470 WatchSource:0}: Error finding container 9bd2408e8eb4a7e2d89ea23a388d9e3cbc5c632044f98564d26ce4f86bd50470: Status 404 returned error can't find the container with id 9bd2408e8eb4a7e2d89ea23a388d9e3cbc5c632044f98564d26ce4f86bd50470 Oct 30 12:58:52.072486 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:52.072454 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:52.072586 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:52.072514 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:52.072586 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:52.072542 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:58:52.196609 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:52.196488 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:52.611368 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:52.611334 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" event={"ID":"25132bf1-77b7-4acb-940e-d36e07a12a76","Type":"ContainerStarted","Data":"34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22"} Oct 30 12:58:52.611544 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:52.611418 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" event={"ID":"25132bf1-77b7-4acb-940e-d36e07a12a76","Type":"ContainerStarted","Data":"9bd2408e8eb4a7e2d89ea23a388d9e3cbc5c632044f98564d26ce4f86bd50470"} Oct 30 12:58:52.674953 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:52.674905 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" podStartSLOduration=1.674892139 podStartE2EDuration="1.674892139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:52.673050974 +0000 UTC m=+1550.799851872" watchObservedRunningTime="2026-10-30 12:58:52.674892139 +0000 UTC m=+1550.801693038" Oct 30 12:58:52.755468 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:52.755419 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:53.754244 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:53.754213 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:58:53.754615 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:53.754282 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:58:53.945305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:53.944902 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:54.707886 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:54.707854 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:54.752553 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:54.752520 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:58:54.752779 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:54.752571 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:58:55.619294 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:55.612723 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:55.753035 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:55.752997 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:58:55.753193 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:55.753064 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:58:56.753260 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:56.753228 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:58:56.753663 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:56.753318 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:58:57.224290 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:57.214999 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:57.752922 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:57.752882 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:58:57.753130 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:57.752939 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:58:58.753408 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:58.753368 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:58:58.753805 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:58.753436 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 12:58:58.899291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:58.892369 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:58.988827 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:58:58.988780 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:58:59.757417 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:59.757390 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:59.757851 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:59.757750 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:58:59.758464 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:58:59.758445 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 12:59:00.488405 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:00.488366 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:02.185329 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:02.169114 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:03.695148 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:03.695115 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:05.826679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:05.826632 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:05.867143 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:05.867039 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:06.095471 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.095360 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:06.110383 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.110338 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:06.137388 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.137348 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:06.162400 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.162347 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:06.663430 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.663396 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4"] Oct 30 12:59:06.738298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.729512 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.960381 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.901895 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.986024 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.986104 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.986143 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.986221 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c06c8e-6813-483b-9ea1-de079980854a-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.986310 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.986336 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c06c8e-6813-483b-9ea1-de079980854a-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.986391 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g59mq\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-kube-api-access-g59mq\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.986475 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.986538 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:06.993312 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:06.988759 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:59:07.086477 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.083374 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4"] Oct 30 12:59:07.112298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.109967 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c06c8e-6813-483b-9ea1-de079980854a-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.112298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.110077 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.112298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.110102 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c06c8e-6813-483b-9ea1-de079980854a-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.112298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.110152 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g59mq\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-kube-api-access-g59mq\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.112298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.110228 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.137293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.126887 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c06c8e-6813-483b-9ea1-de079980854a-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.137293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.127261 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.137293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.128580 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 12:59:07.137293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.129025 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:59:07.184560 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.184523 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 12:59:07.195295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.186092 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:59:07.268346 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.242044 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g59mq\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-kube-api-access-g59mq\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.269651 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.246913 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.328293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.327629 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:07.336291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.329428 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.336291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.329540 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.336291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.329598 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.336291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.329646 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.336291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.331173 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.386864 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.368851 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c06c8e-6813-483b-9ea1-de079980854a-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.396292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.389936 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.518301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.510902 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.589370 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.570459 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:07.589370 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.576161 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.703942 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.703908 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:07.779386 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.779342 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:07.862350 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.861663 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:07.904437 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:07.904351 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:08.158292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.153317 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 12:59:08.175838 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.175808 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:59:08.365427 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.365384 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:08.394298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.390172 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:59:08.463382 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.423467 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:08.573999 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.573942 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:08.575085 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.575064 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:59:08.899376 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.877424 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:08.899376 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.878097 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:59:08.927069 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:08.926363 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:09.040456 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.040411 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:59:09.137547 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.137518 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 12:59:09.140995 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.140974 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:59:09.173352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.168338 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:09.193040 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.192908 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:59:09.204107 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.204070 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 12:59:09.259715 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.259672 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:09.259966 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.259939 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:09.260135 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.260114 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:09.265353 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.265314 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4"] Oct 30 12:59:09.413362 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.413248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:09.641684 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:09.641655 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:10.227264 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:10.227229 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" event={"ID":"52c06c8e-6813-483b-9ea1-de079980854a","Type":"ContainerStarted","Data":"c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a"} Oct 30 12:59:10.227870 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:10.227328 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" event={"ID":"52c06c8e-6813-483b-9ea1-de079980854a","Type":"ContainerStarted","Data":"6ab53fe6d40d0672ce9c36e9ae1f6632d6e47972d3dc83e57516f90b992a8cb2"} Oct 30 12:59:10.288673 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:10.288622 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" podStartSLOduration=4.28860883 podStartE2EDuration="4.28860883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:10.286099559 +0000 UTC m=+1568.412900481" watchObservedRunningTime="2026-10-30 12:59:10.28860883 +0000 UTC m=+1568.415409727" Oct 30 12:59:10.418603 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:10.418554 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:10.633543 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:10.633508 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:10.715292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:10.711554 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:10.718396 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:10.718361 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:59:10.718537 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:10.718424 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:59:10.920340 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:10.918848 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:11.623643 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:11.623591 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:11.709389 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:11.709356 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:59:11.709668 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:11.709640 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:59:11.830617 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:11.830556 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:12.711292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:12.710415 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:59:12.711292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:12.710476 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:59:12.805018 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:12.804984 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:13.715292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:13.714649 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:59:13.715292 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:13.714737 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:59:14.588346 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:14.588308 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:14.721013 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:14.720976 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:59:14.721820 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:14.721786 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:59:15.704631 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:15.704598 5515 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 12:59:15.704841 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:15.704648 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 12:59:16.398296 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:16.393119 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:16.709672 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:16.709599 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:16.709892 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:16.709865 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:16.710595 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:16.710576 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 12:59:18.021585 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:18.021550 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:19.582933 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:19.582686 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:19.944418 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:19.944387 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 12:59:21.341778 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:21.341741 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:22.914401 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:22.914356 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:23.057483 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:23.056354 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:23.100932 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:23.100884 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:23.174503 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:23.174397 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:23.218873 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:23.218834 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:23.245461 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:23.245405 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:24.109295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.102293 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:24.155425 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.155379 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:24.182392 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.179640 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:24.226291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.182585 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:59:24.230448 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.230416 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:59:24.256351 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.251394 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:24.256628 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.256606 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:59:24.285295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.257509 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:59:24.566293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.549548 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:24.846352 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.846209 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:24.897422 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:24.897383 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:25.188308 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:25.162602 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:25.310706 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:25.310649 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:25.363391 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:25.363344 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 12:59:25.477544 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:25.477446 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 12:59:25.831189 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:25.492772 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:25.831189 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:25.552725 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 12:59:25.831189 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:25.793202 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 12:59:25.994294 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:25.979377 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t"] Oct 30 12:59:26.120478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.120389 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.185640 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.185420 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 12:59:26.204208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.190580 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.204208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.190694 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.204208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.190735 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.204208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.190813 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.204208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.190842 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.204208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.190901 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.204208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.190946 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-kube-api-access-2bhxg\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.204208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.191057 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.204208 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.191131 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d01f34-7c20-47f0-8f05-d6c7155febcf-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.294664 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.294624 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.294939 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.294915 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-kube-api-access-2bhxg\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.295136 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.295116 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.299298 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.296817 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.373098 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.372996 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d01f34-7c20-47f0-8f05-d6c7155febcf-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.373436 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.373405 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.377290 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.373677 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.377519 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.377482 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.427380 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.379073 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d01f34-7c20-47f0-8f05-d6c7155febcf-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.449530 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.381033 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-kube-api-access-2bhxg\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.449729 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.387959 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.449867 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.388324 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.449997 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.395492 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.450124 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.419837 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 12:59:26.450301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.422789 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:59:26.450775 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.450263 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t"] Oct 30 12:59:26.456434 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.456408 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.460288 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.456605 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.474458 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.458021 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.477378 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.477355 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 12:59:26.488097 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.487988 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.489904 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.489876 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:26.746234 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:26.745725 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:27.127971 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:27.127936 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t"] Oct 30 12:59:27.128593 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:59:27.128558 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d01f34_7c20_47f0_8f05_d6c7155febcf.slice/crio-6a467516715c89b80e40fbf79e4d47d8a136022d7cb66fe528193f4454ad2537 WatchSource:0}: Error finding container 6a467516715c89b80e40fbf79e4d47d8a136022d7cb66fe528193f4454ad2537: Status 404 returned error can't find the container with id 6a467516715c89b80e40fbf79e4d47d8a136022d7cb66fe528193f4454ad2537 Oct 30 12:59:27.131618 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:27.131586 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:27.131712 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:27.131651 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:27.131712 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:27.131681 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:27.584728 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:27.584695 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:27.802651 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:27.802616 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" event={"ID":"53d01f34-7c20-47f0-8f05-d6c7155febcf","Type":"ContainerStarted","Data":"48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610"} Oct 30 12:59:27.802832 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:27.802675 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" event={"ID":"53d01f34-7c20-47f0-8f05-d6c7155febcf","Type":"ContainerStarted","Data":"6a467516715c89b80e40fbf79e4d47d8a136022d7cb66fe528193f4454ad2537"} Oct 30 12:59:27.880417 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:27.880317 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" podStartSLOduration=2.880301844 podStartE2EDuration="2.880301844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:27.867603299 +0000 UTC m=+1585.994404198" watchObservedRunningTime="2026-10-30 12:59:27.880301844 +0000 UTC m=+1586.007102742" Oct 30 12:59:28.034952 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.034908 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:28.392665 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.392629 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6"] Oct 30 12:59:28.403288 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.401999 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.438690 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.438203 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6"] Oct 30 12:59:28.438690 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.438567 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-87cae1-openshift-default-dockercfg-rhz5h\"" Oct 30 12:59:28.541295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.532928 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-token\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.541295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.532975 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istiod-ca-cert\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.541295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.533059 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-data\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.541295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.533090 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5bqp\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-kube-api-access-x5bqp\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.541295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.533134 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-socket\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.541295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.533393 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-credential-socket\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.541295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.533423 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-envoy\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.541295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.533453 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-podinfo\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.541295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.533510 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-certs\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.639293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639022 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-credential-socket\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.639293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639083 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-envoy\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.639293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639110 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-podinfo\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.639293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639179 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-certs\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.639293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639177 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:28.639293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639301 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-token\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.640141 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639328 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istiod-ca-cert\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.640141 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639443 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-data\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.640141 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639475 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5bqp\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-kube-api-access-x5bqp\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.640141 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.639528 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-socket\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.640370 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.640244 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-socket\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.654336 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.646467 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-credential-socket\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.654336 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.647354 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istiod-ca-cert\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.654336 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.647626 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-certs\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.654336 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.648182 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-data\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.656061 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.656033 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-podinfo\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.692468 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.688332 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-envoy\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.692468 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.692345 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-token\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.715295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.695988 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5bqp\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-kube-api-access-x5bqp\") pod \"e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.746944 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.746905 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:28.771044 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.752981 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:28.771044 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.753099 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:59:28.771044 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:28.753139 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:59:28.989831 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:59:28.989748 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:59:29.140306 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.136677 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:29.191952 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:59:29.191703 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d99f7d_5ea2_45ee_b42d_21bd19c07442.slice/crio-fa01f8f8bac8bd0b8adf0ddef04ac8303f591ab33710b0fa3c9f10d948b50792 WatchSource:0}: Error finding container fa01f8f8bac8bd0b8adf0ddef04ac8303f591ab33710b0fa3c9f10d948b50792: Status 404 returned error can't find the container with id fa01f8f8bac8bd0b8adf0ddef04ac8303f591ab33710b0fa3c9f10d948b50792 Oct 30 12:59:29.220069 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.219937 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6"] Oct 30 12:59:29.227485 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.220421 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:29.227485 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.220592 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:29.227485 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.220697 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:29.439943 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.439904 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:29.746385 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.746352 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:59:29.746769 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.746405 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:59:29.854364 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.854318 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" event={"ID":"b2d99f7d-5ea2-45ee-b42d-21bd19c07442","Type":"ContainerStarted","Data":"e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc"} Oct 30 12:59:29.854551 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.854384 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" event={"ID":"b2d99f7d-5ea2-45ee-b42d-21bd19c07442","Type":"ContainerStarted","Data":"fa01f8f8bac8bd0b8adf0ddef04ac8303f591ab33710b0fa3c9f10d948b50792"} Oct 30 12:59:29.920786 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:29.920704 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" podStartSLOduration=1.920684612 podStartE2EDuration="1.920684612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:29.916355747 +0000 UTC m=+1588.043156645" watchObservedRunningTime="2026-10-30 12:59:29.920684612 +0000 UTC m=+1588.047485513" Oct 30 12:59:30.746395 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:30.746358 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:59:30.746758 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:30.746409 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:59:30.751581 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:30.751561 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:30.756766 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:30.756744 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:30.861390 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:30.861358 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:30.862436 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:30.862413 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 12:59:31.118359 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:31.116937 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:31.746114 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:31.746082 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:59:31.746331 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:31.746214 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:59:32.025806 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:32.025716 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:32.612306 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:32.612130 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:32.746124 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:32.746089 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:59:32.746319 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:32.746138 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:59:32.819696 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:32.819659 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:33.142293 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:33.141223 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:33.746817 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:33.746786 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:59:33.746998 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:33.746835 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 12:59:34.379359 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:34.379291 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:34.690061 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:34.690027 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:34.930872 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:34.930841 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:34.931331 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:34.931301 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:34.932438 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:34.932416 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 12:59:35.112499 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:35.112373 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:36.690845 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:36.690810 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:38.228300 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:38.226601 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:38.430474 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:38.430436 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:38.787255 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:38.787219 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:39.220658 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:39.220624 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:39.922590 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:39.922556 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:40.940025 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:40.939993 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7"] Oct 30 12:59:40.943979 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:40.943960 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:40.979682 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:40.979642 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7"] Oct 30 12:59:41.045955 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.045921 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.045955 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.045954 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.046216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.045990 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.046216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.046042 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.046216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.046120 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43dd811-3ef2-4bf6-aab1-7db158180c51-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.046216 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.046164 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858vs\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-kube-api-access-858vs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.046400 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.046258 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.046400 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.046335 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.046476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.046432 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.147998 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.147963 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148018 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148118 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148142 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148184 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148534 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148301 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148534 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148364 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43dd811-3ef2-4bf6-aab1-7db158180c51-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148534 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148463 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858vs\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-kube-api-access-858vs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148675 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148546 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148675 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148587 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148675 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148629 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148829 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148724 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.148997 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.148972 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.149314 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.149297 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43dd811-3ef2-4bf6-aab1-7db158180c51-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.150438 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.150423 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.150544 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.150528 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.171301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.171235 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.171439 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.171341 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-858vs\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-kube-api-access-858vs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.265067 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.264989 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:41.584214 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.583301 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:41.622955 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.622923 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7"] Oct 30 12:59:41.624667 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:59:41.624636 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43dd811_3ef2_4bf6_aab1_7db158180c51.slice/crio-644c4534ac768f584436bb21402631c91fd32c2c94f41f9090deb0f186fd799b WatchSource:0}: Error finding container 644c4534ac768f584436bb21402631c91fd32c2c94f41f9090deb0f186fd799b: Status 404 returned error can't find the container with id 644c4534ac768f584436bb21402631c91fd32c2c94f41f9090deb0f186fd799b Oct 30 12:59:41.627393 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.627353 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:41.627479 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.627441 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:41.627955 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.627481 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 12:59:41.977110 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.977071 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" event={"ID":"f43dd811-3ef2-4bf6-aab1-7db158180c51","Type":"ContainerStarted","Data":"e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac"} Oct 30 12:59:41.977506 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:41.977130 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" event={"ID":"f43dd811-3ef2-4bf6-aab1-7db158180c51","Type":"ContainerStarted","Data":"644c4534ac768f584436bb21402631c91fd32c2c94f41f9090deb0f186fd799b"} Oct 30 12:59:42.044287 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:42.032192 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" podStartSLOduration=2.032173917 podStartE2EDuration="2.032173917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:42.028855002 +0000 UTC m=+1600.155655902" watchObservedRunningTime="2026-10-30 12:59:42.032173917 +0000 UTC m=+1600.158974816" Oct 30 12:59:42.265504 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:42.265421 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:42.266720 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:42.266695 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:59:42.266849 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:42.266749 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:59:43.073204 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.073173 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr"] Oct 30 12:59:43.077423 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.077403 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.111414 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.111385 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-87cae1-serving-cert\"" Oct 30 12:59:43.133369 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.133329 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr"] Oct 30 12:59:43.177921 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.177888 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-maas-api-tls\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.178106 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.178093 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-openshift-service-ca\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.178151 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.178132 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-metrics-certs\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.178295 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.178245 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6lb\" (UniqueName: \"kubernetes.io/projected/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-kube-api-access-ms6lb\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.266353 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.266319 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:59:43.266537 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.266372 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:59:43.271687 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.271635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:43.279679 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.279655 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6lb\" (UniqueName: \"kubernetes.io/projected/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-kube-api-access-ms6lb\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.279787 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.279692 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-maas-api-tls\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.279855 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.279843 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-openshift-service-ca\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.279899 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.279879 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-metrics-certs\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.280521 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.280501 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-openshift-service-ca\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.282386 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.282363 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-maas-api-tls\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.282460 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.282366 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-metrics-certs\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.322627 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.322590 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6lb\" (UniqueName: \"kubernetes.io/projected/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-kube-api-access-ms6lb\") pod \"maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.395305 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.393628 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:43.512301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.503885 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:43.952515 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.952484 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr"] Oct 30 12:59:43.952633 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:59:43.952562 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b427e7_ed92_46c0_ba0e_e41ce21d2edc.slice/crio-6e378a94af824ac821e6e8db771e44ca7bef186458a6bf5f687445191db4ef75 WatchSource:0}: Error finding container 6e378a94af824ac821e6e8db771e44ca7bef186458a6bf5f687445191db4ef75: Status 404 returned error can't find the container with id 6e378a94af824ac821e6e8db771e44ca7bef186458a6bf5f687445191db4ef75 Oct 30 12:59:43.954103 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.954086 5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:59:43.992317 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:43.992255 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" event={"ID":"02b427e7-ed92-46c0-ba0e-e41ce21d2edc","Type":"ContainerStarted","Data":"6e378a94af824ac821e6e8db771e44ca7bef186458a6bf5f687445191db4ef75"} Oct 30 12:59:44.265608 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:44.265521 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:59:44.265608 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:44.265586 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:59:45.032313 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.031014 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" event={"ID":"02b427e7-ed92-46c0-ba0e-e41ce21d2edc","Type":"ContainerStarted","Data":"ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0"} Oct 30 12:59:45.032313 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.031187 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:45.080451 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.080392 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" podStartSLOduration=1.4867444650000001 podStartE2EDuration="2.080376349s" totalImagesPullingTime="593.631884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:45.076899922 +0000 UTC m=+1603.203700824" watchObservedRunningTime="2026-10-30 12:59:45.080376349 +0000 UTC m=+1603.207177279" Oct 30 12:59:45.231381 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.231349 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z"] Oct 30 12:59:45.235836 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.235817 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:45.253412 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.253388 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ea117f29-dockercfg-sczvb\"" Oct 30 12:59:45.256301 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.254322 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ea117f29\"" Oct 30 12:59:45.258318 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.258297 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z"] Oct 30 12:59:45.266304 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.266262 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:59:45.266620 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.266333 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:59:45.541291 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.534335 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv"] Oct 30 12:59:45.640675 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.634401 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d07376-a4be-404d-8677-16556eae482a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z\" (UID: \"01d07376-a4be-404d-8677-16556eae482a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:45.640675 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.634756 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7bbw\" (UniqueName: \"kubernetes.io/projected/01d07376-a4be-404d-8677-16556eae482a-kube-api-access-v7bbw\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z\" (UID: \"01d07376-a4be-404d-8677-16556eae482a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:45.640675 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.636689 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv"] Oct 30 12:59:45.640675 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.636804 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:45.735624 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.735586 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7bbw\" (UniqueName: \"kubernetes.io/projected/01d07376-a4be-404d-8677-16556eae482a-kube-api-access-v7bbw\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z\" (UID: \"01d07376-a4be-404d-8677-16556eae482a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:45.735624 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.735629 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1def9-8934-459f-b57d-908c661162f9-plugins-config-volume\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv\" (UID: \"8ba1def9-8934-459f-b57d-908c661162f9\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:45.735902 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.735779 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d07376-a4be-404d-8677-16556eae482a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z\" (UID: \"01d07376-a4be-404d-8677-16556eae482a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:45.735902 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.735835 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z957d\" (UniqueName: \"kubernetes.io/projected/8ba1def9-8934-459f-b57d-908c661162f9-kube-api-access-z957d\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv\" (UID: \"8ba1def9-8934-459f-b57d-908c661162f9\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:45.736700 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.736677 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d07376-a4be-404d-8677-16556eae482a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z\" (UID: \"01d07376-a4be-404d-8677-16556eae482a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:45.759318 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.759249 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7bbw\" (UniqueName: \"kubernetes.io/projected/01d07376-a4be-404d-8677-16556eae482a-kube-api-access-v7bbw\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z\" (UID: \"01d07376-a4be-404d-8677-16556eae482a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:45.837063 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.836969 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1def9-8934-459f-b57d-908c661162f9-plugins-config-volume\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv\" (UID: \"8ba1def9-8934-459f-b57d-908c661162f9\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:45.837237 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.837099 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z957d\" (UniqueName: \"kubernetes.io/projected/8ba1def9-8934-459f-b57d-908c661162f9-kube-api-access-z957d\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv\" (UID: \"8ba1def9-8934-459f-b57d-908c661162f9\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:45.837739 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.837711 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1def9-8934-459f-b57d-908c661162f9-plugins-config-volume\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv\" (UID: \"8ba1def9-8934-459f-b57d-908c661162f9\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:45.849300 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.849253 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:45.860766 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.860731 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z957d\" (UniqueName: \"kubernetes.io/projected/8ba1def9-8934-459f-b57d-908c661162f9-kube-api-access-z957d\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv\" (UID: \"8ba1def9-8934-459f-b57d-908c661162f9\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:45.966596 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:45.966556 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:46.182334 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:46.182302 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z"] Oct 30 12:59:46.182852 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:59:46.182823 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d07376_a4be_404d_8677_16556eae482a.slice/crio-7ace092342f6511c0e211b82c1502deef6babf79a78fe5c5e86b71cbf9722b03 WatchSource:0}: Error finding container 7ace092342f6511c0e211b82c1502deef6babf79a78fe5c5e86b71cbf9722b03: Status 404 returned error can't find the container with id 7ace092342f6511c0e211b82c1502deef6babf79a78fe5c5e86b71cbf9722b03 Oct 30 12:59:46.265734 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:46.265704 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:59:46.266398 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:46.265773 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:59:46.322976 ip-10-0-134-193 kubenswrapper[5515]: W1030 12:59:46.322925 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba1def9_8934_459f_b57d_908c661162f9.slice/crio-851be20b5265e9d91221074c89609cc61a95391044adc292b29d0fa7819fbb6e WatchSource:0}: Error finding container 851be20b5265e9d91221074c89609cc61a95391044adc292b29d0fa7819fbb6e: Status 404 returned error can't find the container with id 851be20b5265e9d91221074c89609cc61a95391044adc292b29d0fa7819fbb6e Oct 30 12:59:46.326088 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:46.326059 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv"] Oct 30 12:59:47.048580 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.048550 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:47.050965 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.050935 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" event={"ID":"8ba1def9-8934-459f-b57d-908c661162f9","Type":"ContainerStarted","Data":"44e7d370c9686611ab9ef60488dafea69690f0456607ef454543e13bc8c8aaf7"} Oct 30 12:59:47.051104 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.050996 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" event={"ID":"8ba1def9-8934-459f-b57d-908c661162f9","Type":"ContainerStarted","Data":"851be20b5265e9d91221074c89609cc61a95391044adc292b29d0fa7819fbb6e"} Oct 30 12:59:47.051168 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.051113 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:47.052642 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.052615 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" event={"ID":"01d07376-a4be-404d-8677-16556eae482a","Type":"ContainerStarted","Data":"b53ddf4e9dd54f7ebec99603483395b805e2cbe0ecc44d3d688a34c248816a38"} Oct 30 12:59:47.052759 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.052665 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" event={"ID":"01d07376-a4be-404d-8677-16556eae482a","Type":"ContainerStarted","Data":"7ace092342f6511c0e211b82c1502deef6babf79a78fe5c5e86b71cbf9722b03"} Oct 30 12:59:47.052759 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.052690 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:47.119717 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.119648 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" podStartSLOduration=2.1196266440000002 podStartE2EDuration="2.119626644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:47.112515115 +0000 UTC m=+1605.239316015" watchObservedRunningTime="2026-10-30 12:59:47.119626644 +0000 UTC m=+1605.246427543" Oct 30 12:59:47.171362 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.171280 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" podStartSLOduration=2.171252584 podStartE2EDuration="2.171252584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:47.170062758 +0000 UTC m=+1605.296863657" watchObservedRunningTime="2026-10-30 12:59:47.171252584 +0000 UTC m=+1605.298053510" Oct 30 12:59:47.266239 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.266207 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:59:47.266432 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.266258 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:59:47.283809 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:47.283703 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:48.265447 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:48.265414 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 12:59:48.265660 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:48.265475 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 12:59:48.627662 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:48.611807 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:49.270478 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:49.270450 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:49.270750 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:49.270717 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:49.271445 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:49.271430 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 12:59:50.192407 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:50.192340 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:51.092944 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:51.092902 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 12:59:52.877755 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:52.877722 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:53.213470 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:53.212622 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 12:59:55.525383 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:55.525347 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:57.164929 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:57.164892 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:57.371927 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:57.371894 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:58.141077 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:58.140296 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 12:59:58.141424 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:58.141399 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 12:59:58.890543 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:58.890502 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:58.988701 ip-10-0-134-193 kubenswrapper[5515]: E1030 12:59:58.988662 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 12:59:59.670476 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:59.670438 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:59.932783 ip-10-0-134-193 kubenswrapper[5515]: I1030 12:59:59.932703 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:00.129911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.129879 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5"] Oct 30 13:00:00.134233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.134211 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.176399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.176367 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2e0e28-openshift-default-dockercfg-rjb94\"" Oct 30 13:00:00.182302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.182262 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5"] Oct 30 13:00:00.277355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.239650 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-socket\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.361600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.361557 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istiod-ca-cert\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.361947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.361927 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-credential-socket\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.362123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.362105 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdczg\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-kube-api-access-bdczg\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.362228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.362215 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-token\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.378004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.377970 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-podinfo\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.378353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.378328 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-certs\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.386304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.378500 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-envoy\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.527523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.512837 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-data\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.527523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.431142 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6"] Oct 30 13:00:00.586646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.586613 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps"] Oct 30 13:00:00.602435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.602355 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:00.635644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.635595 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6"] Oct 30 13:00:00.638365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.638334 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:00.648969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.648031 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-token\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.648969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.648170 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-envoy\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.648969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.648261 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-podinfo\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.648969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.648493 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9bh\" (UniqueName: \"kubernetes.io/projected/9224217c-166c-4a45-a13a-7b4bfffe8011-kube-api-access-4l9bh\") pod \"maas-api-key-cleanup-29889420-c94v6\" (UID: \"9224217c-166c-4a45-a13a-7b4bfffe8011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:00.648969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.648629 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-credential-socket\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.648969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.648665 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9224217c-166c-4a45-a13a-7b4bfffe8011-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-c94v6\" (UID: \"9224217c-166c-4a45-a13a-7b4bfffe8011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:00.648969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.648708 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdczg\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-kube-api-access-bdczg\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.648969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.648967 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-certs\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.649537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.648979 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\"" Oct 30 13:00:00.649537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.649036 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-data\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.649537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.649116 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-socket\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.649537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.649145 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istiod-ca-cert\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.651222 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.651184 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istiod-ca-cert\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.662364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.662055 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-data\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.662364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.662304 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-certs\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.662681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.662655 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-socket\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.663014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.662974 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-credential-socket\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.664579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.664556 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps"] Oct 30 13:00:00.665012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.664991 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-envoy\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.672473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.667440 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-podinfo\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.675386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.675352 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:00.685496 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.685466 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-token\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.698282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.698221 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdczg\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-kube-api-access-bdczg\") pod \"e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.748881 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.748848 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:00.751682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.751657 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9bh\" (UniqueName: \"kubernetes.io/projected/9224217c-166c-4a45-a13a-7b4bfffe8011-kube-api-access-4l9bh\") pod \"maas-api-key-cleanup-29889420-c94v6\" (UID: \"9224217c-166c-4a45-a13a-7b4bfffe8011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:00.751826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.751747 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9224217c-166c-4a45-a13a-7b4bfffe8011-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-c94v6\" (UID: \"9224217c-166c-4a45-a13a-7b4bfffe8011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:00.751826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.751819 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64c6j\" (UniqueName: \"kubernetes.io/projected/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-kube-api-access-64c6j\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps\" (UID: \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:00.751930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.751889 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps\" (UID: \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:00.752610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.752592 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9224217c-166c-4a45-a13a-7b4bfffe8011-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889420-c94v6\" (UID: \"9224217c-166c-4a45-a13a-7b4bfffe8011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:00.775122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.775086 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9bh\" (UniqueName: \"kubernetes.io/projected/9224217c-166c-4a45-a13a-7b4bfffe8011-kube-api-access-4l9bh\") pod \"maas-api-key-cleanup-29889420-c94v6\" (UID: \"9224217c-166c-4a45-a13a-7b4bfffe8011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:00.866372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.865787 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64c6j\" (UniqueName: \"kubernetes.io/projected/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-kube-api-access-64c6j\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps\" (UID: \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:00.866372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.865907 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps\" (UID: \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:00.872816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.869175 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps\" (UID: \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:00.889355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.889314 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:00.900747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:00.895769 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64c6j\" (UniqueName: \"kubernetes.io/projected/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-kube-api-access-64c6j\") pod \"maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps\" (UID: \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:01.030792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.030758 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:01.036175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.036148 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:01.148678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.148450 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5"] Oct 30 13:00:01.150252 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:00:01.150223 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffad291_ae66_47ba_9b0f_f1aeb0ff424e.slice/crio-ce2814572afc3c7938cda903d574059e4ae203c18da23caf3048bb4ee52aea25 WatchSource:0}: Error finding container ce2814572afc3c7938cda903d574059e4ae203c18da23caf3048bb4ee52aea25: Status 404 returned error can't find the container with id ce2814572afc3c7938cda903d574059e4ae203c18da23caf3048bb4ee52aea25 Oct 30 13:00:01.153450 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.153404 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:01.153541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.153508 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:01.153590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.153553 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:01.337173 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:00:01.337103 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9224217c_166c_4a45_a13a_7b4bfffe8011.slice/crio-4f1584edf8233c8f19c04f713c5c023d60da4f91cabd249aed9a8b0abe2352f4 WatchSource:0}: Error finding container 4f1584edf8233c8f19c04f713c5c023d60da4f91cabd249aed9a8b0abe2352f4: Status 404 returned error can't find the container with id 4f1584edf8233c8f19c04f713c5c023d60da4f91cabd249aed9a8b0abe2352f4 Oct 30 13:00:01.337527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.337499 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6"] Oct 30 13:00:01.350748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.346718 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps"] Oct 30 13:00:01.352529 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:00:01.352498 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc8e9ae_a156_43c2_8039_5d3d1b18b7eb.slice/crio-51fffcbb02bb30a378084cac6174e3a34aacf807346b4339ab8ba9d14521fba8 WatchSource:0}: Error finding container 51fffcbb02bb30a378084cac6174e3a34aacf807346b4339ab8ba9d14521fba8: Status 404 returned error can't find the container with id 51fffcbb02bb30a378084cac6174e3a34aacf807346b4339ab8ba9d14521fba8 Oct 30 13:00:01.706722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.706689 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" event={"ID":"9224217c-166c-4a45-a13a-7b4bfffe8011","Type":"ContainerStarted","Data":"4f1584edf8233c8f19c04f713c5c023d60da4f91cabd249aed9a8b0abe2352f4"} Oct 30 13:00:01.708969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.708946 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" event={"ID":"affad291-ae66-47ba-9b0f-f1aeb0ff424e","Type":"ContainerStarted","Data":"fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4"} Oct 30 13:00:01.709073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.708993 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" event={"ID":"affad291-ae66-47ba-9b0f-f1aeb0ff424e","Type":"ContainerStarted","Data":"ce2814572afc3c7938cda903d574059e4ae203c18da23caf3048bb4ee52aea25"} Oct 30 13:00:01.711794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.711768 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" event={"ID":"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb","Type":"ContainerStarted","Data":"51fffcbb02bb30a378084cac6174e3a34aacf807346b4339ab8ba9d14521fba8"} Oct 30 13:00:01.749905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.749860 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:01.763675 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:01.763619 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" podStartSLOduration=1.763604323 podStartE2EDuration="1.763604323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:01.762687786 +0000 UTC m=+1619.889488684" watchObservedRunningTime="2026-10-30 13:00:01.763604323 +0000 UTC m=+1619.890405221" Oct 30 13:00:02.249210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.249180 5515 scope.go:122] "RemoveContainer" containerID="b52aa46b10124c59ccc41c420a03f3b388ed14d5fbd7ad176f413d8037ecaccb" Oct 30 13:00:02.738287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.733084 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:02.746216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.746175 5515 generic.go:356] "Generic (PLEG): container finished" podID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerID="99a82430df1373735a4e1df0dcb641607b3c385e6f3638a96c95863f10c06310" exitCode=60 Oct 30 13:00:02.746403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.746288 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" event={"ID":"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb","Type":"ContainerDied","Data":"99a82430df1373735a4e1df0dcb641607b3c385e6f3638a96c95863f10c06310"} Oct 30 13:00:02.746642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.746623 5515 scope.go:122] "RemoveContainer" containerID="99a82430df1373735a4e1df0dcb641607b3c385e6f3638a96c95863f10c06310" Oct 30 13:00:02.748113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.748087 5515 generic.go:356] "Generic (PLEG): container finished" podID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerID="9243c7450f05cfd3a3aba163b2884323ec557a1a340e885dbe31e502ed1e4611" exitCode=60 Oct 30 13:00:02.748221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.748161 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" event={"ID":"9224217c-166c-4a45-a13a-7b4bfffe8011","Type":"ContainerDied","Data":"9243c7450f05cfd3a3aba163b2884323ec557a1a340e885dbe31e502ed1e4611"} Oct 30 13:00:02.748501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.748479 5515 scope.go:122] "RemoveContainer" containerID="9243c7450f05cfd3a3aba163b2884323ec557a1a340e885dbe31e502ed1e4611" Oct 30 13:00:02.754738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.754717 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:02.947347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:02.943831 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:03.663029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.662995 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:03.756966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.756928 5515 generic.go:356] "Generic (PLEG): container finished" podID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerID="cfa7749c0b76fa325eb0bb38831ad6d4a4039456b7e6e69ca46fe869094f6f7a" exitCode=60 Oct 30 13:00:03.757151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.757010 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" event={"ID":"9224217c-166c-4a45-a13a-7b4bfffe8011","Type":"ContainerDied","Data":"cfa7749c0b76fa325eb0bb38831ad6d4a4039456b7e6e69ca46fe869094f6f7a"} Oct 30 13:00:03.757151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.757069 5515 scope.go:122] "RemoveContainer" containerID="9243c7450f05cfd3a3aba163b2884323ec557a1a340e885dbe31e502ed1e4611" Oct 30 13:00:03.757381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.757356 5515 scope.go:122] "RemoveContainer" containerID="cfa7749c0b76fa325eb0bb38831ad6d4a4039456b7e6e69ca46fe869094f6f7a" Oct 30 13:00:03.757757 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:00:03.757737 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-c94v6_odh-ai-gateway-infra(9224217c-166c-4a45-a13a-7b4bfffe8011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" Oct 30 13:00:03.759151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.759125 5515 generic.go:356] "Generic (PLEG): container finished" podID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerID="879fc89e4adc6815e70ee49ac1e094817a7ae4248704e0bb8e33bd795796b4c2" exitCode=60 Oct 30 13:00:03.759255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.759205 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" event={"ID":"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb","Type":"ContainerDied","Data":"879fc89e4adc6815e70ee49ac1e094817a7ae4248704e0bb8e33bd795796b4c2"} Oct 30 13:00:03.759523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.759506 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:03.759644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.759623 5515 scope.go:122] "RemoveContainer" containerID="879fc89e4adc6815e70ee49ac1e094817a7ae4248704e0bb8e33bd795796b4c2" Oct 30 13:00:03.760049 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:00:03.759968 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps_odh-ai-gateway-infra(bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" Oct 30 13:00:03.760708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.760690 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:03.822276 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:03.822242 5515 scope.go:122] "RemoveContainer" containerID="99a82430df1373735a4e1df0dcb641607b3c385e6f3638a96c95863f10c06310" Oct 30 13:00:04.127349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.126976 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4"] Oct 30 13:00:04.185756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.182698 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.198680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.198412 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f399fe-openshift-default-dockercfg-hn29w\"" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.294195 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-token\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.307305 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-envoy\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.307343 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-data\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.307368 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-socket\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.307435 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f8ce64-4c88-4269-ab23-c96097bd9669-istiod-ca-cert\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.307521 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-credential-socket\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.307588 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-certs\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.307636 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhz7x\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-kube-api-access-fhz7x\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.307666 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-podinfo\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.308195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.299672 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4"] Oct 30 13:00:04.319876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.319205 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:04.409109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409073 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-token\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409120 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-envoy\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409161 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-data\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409189 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-socket\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409243 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f8ce64-4c88-4269-ab23-c96097bd9669-istiod-ca-cert\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409366 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-credential-socket\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409437 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-certs\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409501 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhz7x\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-kube-api-access-fhz7x\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409544 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-podinfo\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409674 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-data\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409721 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-socket\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409848 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-credential-socket\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.409886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.409873 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-certs\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.410089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.410073 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f8ce64-4c88-4269-ab23-c96097bd9669-istiod-ca-cert\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.411810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.411790 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-envoy\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.411899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.411866 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-podinfo\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.432874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.432837 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-token\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.433094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.433042 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhz7x\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-kube-api-access-fhz7x\") pod \"e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.551293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.550958 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:04.551483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.551447 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:04.773394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.773363 5515 scope.go:122] "RemoveContainer" containerID="cfa7749c0b76fa325eb0bb38831ad6d4a4039456b7e6e69ca46fe869094f6f7a" Oct 30 13:00:04.773806 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:00:04.773778 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-c94v6_odh-ai-gateway-infra(9224217c-166c-4a45-a13a-7b4bfffe8011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" Oct 30 13:00:04.777754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:04.777728 5515 scope.go:122] "RemoveContainer" containerID="879fc89e4adc6815e70ee49ac1e094817a7ae4248704e0bb8e33bd795796b4c2" Oct 30 13:00:04.778102 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:00:04.778071 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps_odh-ai-gateway-infra(bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" Oct 30 13:00:05.027363 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:00:05.027332 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f8ce64_4c88_4269_ab23_c96097bd9669.slice/crio-417f543db47bffc0d6dd9b2aa7256c2525a2314733ce20b5ef891c68a0d0bb00 WatchSource:0}: Error finding container 417f543db47bffc0d6dd9b2aa7256c2525a2314733ce20b5ef891c68a0d0bb00: Status 404 returned error can't find the container with id 417f543db47bffc0d6dd9b2aa7256c2525a2314733ce20b5ef891c68a0d0bb00 Oct 30 13:00:05.028175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:05.028151 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4"] Oct 30 13:00:05.029951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:05.029920 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:05.030008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:05.029984 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:05.030040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:05.030016 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:05.794255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:05.794209 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" event={"ID":"57f8ce64-4c88-4269-ab23-c96097bd9669","Type":"ContainerStarted","Data":"489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1"} Oct 30 13:00:05.794754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:05.794333 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" event={"ID":"57f8ce64-4c88-4269-ab23-c96097bd9669","Type":"ContainerStarted","Data":"417f543db47bffc0d6dd9b2aa7256c2525a2314733ce20b5ef891c68a0d0bb00"} Oct 30 13:00:05.864204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:05.864126 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" podStartSLOduration=1.8641102520000001 podStartE2EDuration="1.864110252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:05.851507833 +0000 UTC m=+1623.978308745" watchObservedRunningTime="2026-10-30 13:00:05.864110252 +0000 UTC m=+1623.990911202" Oct 30 13:00:06.052835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:06.052753 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:06.363337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:06.362305 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:06.573309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:06.564332 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:06.582256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:06.579371 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:06.611292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:06.597765 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:06.801521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:06.801489 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:06.802479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:06.802462 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:08.123109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:08.123079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:08.589988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:08.589957 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:08.795590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:08.795555 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:09.422176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:09.422142 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:09.800202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:09.800066 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:10.313678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:10.313641 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:10.549482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:10.549331 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:11.569081 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:11.569049 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:13.196356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:13.188808 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:14.675081 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:14.675038 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:00:14.844292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:14.835750 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:14.994381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:14.994204 5515 scope.go:122] "RemoveContainer" containerID="879fc89e4adc6815e70ee49ac1e094817a7ae4248704e0bb8e33bd795796b4c2" Oct 30 13:00:16.050294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:16.038788 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" event={"ID":"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb","Type":"ContainerStarted","Data":"fa84d402944eddbaed9066b7d35c467599caea72acfaa8759d3ee5af0ff85f76"} Oct 30 13:00:16.125093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:16.118475 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" podStartSLOduration=16.118450061 podStartE2EDuration="16.118450061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:16.0816917 +0000 UTC m=+1634.208492642" watchObservedRunningTime="2026-10-30 13:00:16.118450061 +0000 UTC m=+1634.245250960" Oct 30 13:00:16.268156 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:00:16.268111 5515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc8e9ae_a156_43c2_8039_5d3d1b18b7eb.slice/crio-conmon-fa84d402944eddbaed9066b7d35c467599caea72acfaa8759d3ee5af0ff85f76.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc8e9ae_a156_43c2_8039_5d3d1b18b7eb.slice/crio-fa84d402944eddbaed9066b7d35c467599caea72acfaa8759d3ee5af0ff85f76.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:00:16.753297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:16.750849 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:17.060348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:17.059520 5515 generic.go:356] "Generic (PLEG): container finished" podID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerID="fa84d402944eddbaed9066b7d35c467599caea72acfaa8759d3ee5af0ff85f76" exitCode=60 Oct 30 13:00:17.060348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:17.059759 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" event={"ID":"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb","Type":"ContainerDied","Data":"fa84d402944eddbaed9066b7d35c467599caea72acfaa8759d3ee5af0ff85f76"} Oct 30 13:00:17.060348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:17.059825 5515 scope.go:122] "RemoveContainer" containerID="879fc89e4adc6815e70ee49ac1e094817a7ae4248704e0bb8e33bd795796b4c2" Oct 30 13:00:17.060980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:17.060380 5515 scope.go:122] "RemoveContainer" containerID="fa84d402944eddbaed9066b7d35c467599caea72acfaa8759d3ee5af0ff85f76" Oct 30 13:00:17.060980 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:00:17.060777 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps_odh-ai-gateway-infra(bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" Oct 30 13:00:17.113398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:17.113350 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps"] Oct 30 13:00:18.016294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.003306 5515 scope.go:122] "RemoveContainer" containerID="cfa7749c0b76fa325eb0bb38831ad6d4a4039456b7e6e69ca46fe869094f6f7a" Oct 30 13:00:18.329614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.329588 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:18.420397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.420358 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-kube-api-access-64c6j\" (UniqueName: \"kubernetes.io/projected/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-kube-api-access-64c6j\") pod \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\" (UID: \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\") " Oct 30 13:00:18.426525 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.426488 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-maas-api-service-ca\") pod \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\" (UID: \"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb\") " Oct 30 13:00:18.426908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.426884 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-maas-api-service-ca" pod "bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" (UID: "bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:18.427317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.427296 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:00:18.430780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.430746 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-kube-api-access-64c6j" pod "bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" (UID: "bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb"). InnerVolumeSpecName "kube-api-access-64c6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:18.518307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.516780 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:18.529428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.529397 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64c6j\" (UniqueName: \"kubernetes.io/projected/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb-kube-api-access-64c6j\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:00:18.744229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:18.744153 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:19.016293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:19.010737 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:19.077093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:19.077063 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" Oct 30 13:00:19.077311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:19.077102 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps" event={"ID":"bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb","Type":"ContainerDied","Data":"51fffcbb02bb30a378084cac6174e3a34aacf807346b4339ab8ba9d14521fba8"} Oct 30 13:00:19.077311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:19.077160 5515 scope.go:122] "RemoveContainer" containerID="fa84d402944eddbaed9066b7d35c467599caea72acfaa8759d3ee5af0ff85f76" Oct 30 13:00:19.079443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:19.079418 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" event={"ID":"9224217c-166c-4a45-a13a-7b4bfffe8011","Type":"ContainerStarted","Data":"998ebd889c091176596b33267bb02e8b91984efe2224a21540003d3ecdf7438b"} Oct 30 13:00:19.147206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:19.147148 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" podStartSLOduration=19.147134192 podStartE2EDuration="19.147134192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:19.144114756 +0000 UTC m=+1637.270915655" watchObservedRunningTime="2026-10-30 13:00:19.147134192 +0000 UTC m=+1637.273935089" Oct 30 13:00:19.183921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:19.183888 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps"] Oct 30 13:00:19.220342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:19.220301 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-551dd1-29889420-zt8ps"] Oct 30 13:00:20.038114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:20.038081 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" path="/var/lib/kubelet/pods/bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb/volumes" Oct 30 13:00:20.088840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:20.088808 5515 generic.go:356] "Generic (PLEG): container finished" podID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerID="998ebd889c091176596b33267bb02e8b91984efe2224a21540003d3ecdf7438b" exitCode=60 Oct 30 13:00:20.089010 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:20.088864 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" event={"ID":"9224217c-166c-4a45-a13a-7b4bfffe8011","Type":"ContainerDied","Data":"998ebd889c091176596b33267bb02e8b91984efe2224a21540003d3ecdf7438b"} Oct 30 13:00:20.089010 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:20.088913 5515 scope.go:122] "RemoveContainer" containerID="cfa7749c0b76fa325eb0bb38831ad6d4a4039456b7e6e69ca46fe869094f6f7a" Oct 30 13:00:20.089310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:20.089288 5515 scope.go:122] "RemoveContainer" containerID="998ebd889c091176596b33267bb02e8b91984efe2224a21540003d3ecdf7438b" Oct 30 13:00:20.089633 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:00:20.089608 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889420-c94v6_odh-ai-gateway-infra(9224217c-166c-4a45-a13a-7b4bfffe8011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" Oct 30 13:00:20.154689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:20.154653 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6"] Oct 30 13:00:20.466647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:20.466542 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:21.329693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:21.329660 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:21.362025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:21.361993 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9224217c-166c-4a45-a13a-7b4bfffe8011-kube-api-access-4l9bh\" (UniqueName: \"kubernetes.io/projected/9224217c-166c-4a45-a13a-7b4bfffe8011-kube-api-access-4l9bh\") pod \"9224217c-166c-4a45-a13a-7b4bfffe8011\" (UID: \"9224217c-166c-4a45-a13a-7b4bfffe8011\") " Oct 30 13:00:21.362212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:21.362116 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9224217c-166c-4a45-a13a-7b4bfffe8011-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9224217c-166c-4a45-a13a-7b4bfffe8011-maas-api-service-ca\") pod \"9224217c-166c-4a45-a13a-7b4bfffe8011\" (UID: \"9224217c-166c-4a45-a13a-7b4bfffe8011\") " Oct 30 13:00:21.362552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:21.362519 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9224217c-166c-4a45-a13a-7b4bfffe8011-maas-api-service-ca" pod "9224217c-166c-4a45-a13a-7b4bfffe8011" (UID: "9224217c-166c-4a45-a13a-7b4bfffe8011"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:21.362716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:21.362703 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9224217c-166c-4a45-a13a-7b4bfffe8011-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.364123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:21.364089 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9224217c-166c-4a45-a13a-7b4bfffe8011-kube-api-access-4l9bh" pod "9224217c-166c-4a45-a13a-7b4bfffe8011" (UID: "9224217c-166c-4a45-a13a-7b4bfffe8011"). InnerVolumeSpecName "kube-api-access-4l9bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:21.463811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:21.463778 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l9bh\" (UniqueName: \"kubernetes.io/projected/9224217c-166c-4a45-a13a-7b4bfffe8011-kube-api-access-4l9bh\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:00:22.143293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.135626 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" event={"ID":"9224217c-166c-4a45-a13a-7b4bfffe8011","Type":"ContainerDied","Data":"4f1584edf8233c8f19c04f713c5c023d60da4f91cabd249aed9a8b0abe2352f4"} Oct 30 13:00:22.143293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.135706 5515 scope.go:122] "RemoveContainer" containerID="998ebd889c091176596b33267bb02e8b91984efe2224a21540003d3ecdf7438b" Oct 30 13:00:22.143293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.135818 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6" Oct 30 13:00:22.186383 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.186348 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:22.252478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.250319 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6"] Oct 30 13:00:22.261293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.254759 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889420-c94v6"] Oct 30 13:00:22.370000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.369923 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf"] Oct 30 13:00:22.370633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370613 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerName="cleanup" Oct 30 13:00:22.370633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370635 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerName="cleanup" Oct 30 13:00:22.370751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370668 5515 cpu_manager.go:414] "containerMap: removing container" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerName="cleanup" Oct 30 13:00:22.370751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370673 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerName="cleanup" Oct 30 13:00:22.370751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370683 5515 cpu_manager.go:414] "containerMap: removing container" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerName="cleanup" Oct 30 13:00:22.370751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370688 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerName="cleanup" Oct 30 13:00:22.370751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370705 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerName="cleanup" Oct 30 13:00:22.370751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370710 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerName="cleanup" Oct 30 13:00:22.370751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370722 5515 cpu_manager.go:414] "containerMap: removing container" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerName="cleanup" Oct 30 13:00:22.370751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370727 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerName="cleanup" Oct 30 13:00:22.371116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370825 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerName="cleanup" Oct 30 13:00:22.371116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370836 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerName="cleanup" Oct 30 13:00:22.371116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370843 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerName="cleanup" Oct 30 13:00:22.371116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370859 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerName="cleanup" Oct 30 13:00:22.371116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.370867 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc8e9ae-a156-43c2-8039-5d3d1b18b7eb" containerName="cleanup" Oct 30 13:00:22.375630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.375607 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.389004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.388981 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-tmp-dir\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.389147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.389109 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-home\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.389217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.389196 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97c75bb-9fae-4978-a460-39b24d898ad5-tls-certs\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.389346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.389325 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-model-cache\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.389509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.389479 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzcz5\" (UniqueName: \"kubernetes.io/projected/a97c75bb-9fae-4978-a460-39b24d898ad5-kube-api-access-hzcz5\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.389568 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.389534 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-dshm\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.392945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.392920 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ea117f29\"/\"default-dockercfg-d8x2v\"" Oct 30 13:00:22.412313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.411461 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ea117f29\"/\"openshift-service-ca.crt\"" Oct 30 13:00:22.412313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.411645 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ea117f29\"/\"e2e-lifecycle-model-ea117f29-kserve-self-signed-certs\"" Oct 30 13:00:22.412313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.411726 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ea117f29\"/\"kube-root-ca.crt\"" Oct 30 13:00:22.434868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.430740 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf"] Oct 30 13:00:22.490538 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.490499 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-dshm\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.490720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.490565 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-tmp-dir\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.490720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.490626 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-home\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.490720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.490680 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97c75bb-9fae-4978-a460-39b24d898ad5-tls-certs\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.490898 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.490754 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-model-cache\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.490962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.490915 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcz5\" (UniqueName: \"kubernetes.io/projected/a97c75bb-9fae-4978-a460-39b24d898ad5-kube-api-access-hzcz5\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.491403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.491371 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-home\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.491403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.491387 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-tmp-dir\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.491608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.491451 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-model-cache\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.501295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.493453 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-dshm\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.522190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.522129 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97c75bb-9fae-4978-a460-39b24d898ad5-tls-certs\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.525487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.525457 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:22.526115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.526091 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcz5\" (UniqueName: \"kubernetes.io/projected/a97c75bb-9fae-4978-a460-39b24d898ad5-kube-api-access-hzcz5\") pod \"e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:22.693298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:22.690651 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:23.146244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:23.146219 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf"] Oct 30 13:00:23.147360 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:00:23.147339 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda97c75bb_9fae_4978_a460_39b24d898ad5.slice/crio-ab8d540c22809ac59852589dc75ea6a15a04a0b75a6e3a3a73e6925efea65ce5 WatchSource:0}: Error finding container ab8d540c22809ac59852589dc75ea6a15a04a0b75a6e3a3a73e6925efea65ce5: Status 404 returned error can't find the container with id ab8d540c22809ac59852589dc75ea6a15a04a0b75a6e3a3a73e6925efea65ce5 Oct 30 13:00:23.856918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:23.856880 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:24.012950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:24.012915 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" path="/var/lib/kubelet/pods/9224217c-166c-4a45-a13a-7b4bfffe8011/volumes" Oct 30 13:00:24.063339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:24.063302 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:24.153797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:24.153764 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" event={"ID":"a97c75bb-9fae-4978-a460-39b24d898ad5","Type":"ContainerStarted","Data":"a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be"} Oct 30 13:00:24.153977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:24.153807 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" event={"ID":"a97c75bb-9fae-4978-a460-39b24d898ad5","Type":"ContainerStarted","Data":"ab8d540c22809ac59852589dc75ea6a15a04a0b75a6e3a3a73e6925efea65ce5"} Oct 30 13:00:24.207657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:24.207597 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" podStartSLOduration=2.207582719 podStartE2EDuration="2.207582719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:24.204064599 +0000 UTC m=+1642.330865511" watchObservedRunningTime="2026-10-30 13:00:24.207582719 +0000 UTC m=+1642.334383616" Oct 30 13:00:24.691395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:24.691365 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:25.611308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:25.603844 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:27.188327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:27.188243 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:27.225362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:27.225237 5515 generic.go:356] "Generic (PLEG): container finished" podID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerID="e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc" exitCode=0 Oct 30 13:00:27.225583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:27.225363 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" event={"ID":"b2d99f7d-5ea2-45ee-b42d-21bd19c07442","Type":"ContainerDied","Data":"e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc"} Oct 30 13:00:27.225873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:27.225856 5515 scope.go:122] "RemoveContainer" containerID="e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc" Oct 30 13:00:27.227848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:27.227808 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:27.227985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:27.227905 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:27.227985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:27.227954 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:28.234074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:28.234038 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" event={"ID":"b2d99f7d-5ea2-45ee-b42d-21bd19c07442","Type":"ContainerStarted","Data":"ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29"} Oct 30 13:00:28.751638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:28.751600 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 13:00:28.752904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:28.752882 5515 patch_prober.go:29] interesting pod/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:00:28.752970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:28.752938 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:00:28.933294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:28.926234 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:29.039192 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:00:29.039110 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:00:29.752547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:29.752517 5515 patch_prober.go:29] interesting pod/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:00:29.752969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:29.752574 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:00:30.598744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:30.598690 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:30.752588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:30.752554 5515 patch_prober.go:29] interesting pod/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:00:30.752998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:30.752603 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:00:30.862035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:30.861964 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 13:00:31.751579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:31.751549 5515 patch_prober.go:29] interesting pod/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:00:31.751764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:31.751596 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:00:32.253301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:32.250828 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:32.692067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:32.692038 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:32.735467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:32.735431 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:32.752372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:32.752342 5515 patch_prober.go:29] interesting pod/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 13:00:32.752512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:32.752388 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 13:00:33.287923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:33.287887 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:00:33.649329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:33.649295 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:33.824324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:33.817835 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 13:00:33.824324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:33.819782 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 13:00:35.395304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:35.395241 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:36.942223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:36.942184 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:38.523293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:38.497799 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:39.778292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:39.778242 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:40.008512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:40.008461 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:40.450843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:40.450715 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:40.926877 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:40.926805 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:41.141448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:41.132717 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:41.595936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:41.595811 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:43.301822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:43.301771 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:44.120439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:44.120400 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:45.095955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:45.095923 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:45.299534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:45.299412 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:45.467699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:45.467597 5515 generic.go:356] "Generic (PLEG): container finished" podID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerID="489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1" exitCode=0 Oct 30 13:00:45.467814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:45.467789 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" event={"ID":"57f8ce64-4c88-4269-ab23-c96097bd9669","Type":"ContainerDied","Data":"489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1"} Oct 30 13:00:45.468230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:45.468207 5515 scope.go:122] "RemoveContainer" containerID="489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1" Oct 30 13:00:45.470012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:45.469978 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:45.470111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:45.470066 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:45.470178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:45.470118 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:46.476685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.476653 5515 generic.go:356] "Generic (PLEG): container finished" podID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerID="fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4" exitCode=0 Oct 30 13:00:46.477182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.476730 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" event={"ID":"affad291-ae66-47ba-9b0f-f1aeb0ff424e","Type":"ContainerDied","Data":"fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4"} Oct 30 13:00:46.477303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.477251 5515 scope.go:122] "RemoveContainer" containerID="fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4" Oct 30 13:00:46.478896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.478859 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:46.479021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.478949 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:46.479021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.478999 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:00:46.479165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.479140 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" event={"ID":"57f8ce64-4c88-4269-ab23-c96097bd9669","Type":"ContainerStarted","Data":"0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a"} Oct 30 13:00:46.552054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.552023 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:46.552375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.552355 5515 patch_prober.go:29] interesting pod/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:00:46.552494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.552414 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 13:00:46.850521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.850424 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:46.963250 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.963207 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk"] Oct 30 13:00:46.963997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.963973 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerName="cleanup" Oct 30 13:00:46.963997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.963996 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerName="cleanup" Oct 30 13:00:46.964192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.964178 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9224217c-166c-4a45-a13a-7b4bfffe8011" containerName="cleanup" Oct 30 13:00:46.976689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:46.976660 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:47.047296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.042975 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk"] Oct 30 13:00:47.069338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.048534 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk\" (UID: \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:47.083303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.077560 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm664\" (UniqueName: \"kubernetes.io/projected/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-kube-api-access-cm664\") pod \"payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk\" (UID: \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:47.242296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.171995 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4561d2-dockercfg-jqf2q\"" Oct 30 13:00:47.242296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.172078 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4561d2\"" Oct 30 13:00:47.242296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.182489 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd"] Oct 30 13:00:47.242296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.198221 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:47.246186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.246156 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:47.287035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.282146 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk\" (UID: \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:47.287035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.282655 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm664\" (UniqueName: \"kubernetes.io/projected/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-kube-api-access-cm664\") pod \"payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk\" (UID: \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:47.287035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.282707 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e89920-62df-4684-8d87-a73b607cf8e3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd\" (UID: \"45e89920-62df-4684-8d87-a73b607cf8e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:47.287035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.282830 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwv2s\" (UniqueName: \"kubernetes.io/projected/45e89920-62df-4684-8d87-a73b607cf8e3-kube-api-access-nwv2s\") pod \"payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd\" (UID: \"45e89920-62df-4684-8d87-a73b607cf8e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:47.287035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.284987 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk\" (UID: \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:47.345293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.343376 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd"] Oct 30 13:00:47.346147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.346120 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm664\" (UniqueName: \"kubernetes.io/projected/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-kube-api-access-cm664\") pod \"payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk\" (UID: \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:47.384844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.384804 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e89920-62df-4684-8d87-a73b607cf8e3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd\" (UID: \"45e89920-62df-4684-8d87-a73b607cf8e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:47.385032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.384919 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwv2s\" (UniqueName: \"kubernetes.io/projected/45e89920-62df-4684-8d87-a73b607cf8e3-kube-api-access-nwv2s\") pod \"payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd\" (UID: \"45e89920-62df-4684-8d87-a73b607cf8e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:47.385563 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.385542 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e89920-62df-4684-8d87-a73b607cf8e3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd\" (UID: \"45e89920-62df-4684-8d87-a73b607cf8e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:47.423291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.423127 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwv2s\" (UniqueName: \"kubernetes.io/projected/45e89920-62df-4684-8d87-a73b607cf8e3-kube-api-access-nwv2s\") pod \"payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd\" (UID: \"45e89920-62df-4684-8d87-a73b607cf8e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:47.486171 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.486137 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:47.491602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.491558 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" event={"ID":"affad291-ae66-47ba-9b0f-f1aeb0ff424e","Type":"ContainerStarted","Data":"054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72"} Oct 30 13:00:47.552650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.552178 5515 patch_prober.go:29] interesting pod/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:00:47.552650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.552240 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 13:00:47.654033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.653749 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:47.749476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.749244 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:47.752424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.752392 5515 patch_prober.go:29] interesting pod/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:00:47.752572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.752449 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 13:00:47.952314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:47.952282 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk"] Oct 30 13:00:47.953061 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:00:47.953020 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb1cb3f_909e_4f38_b293_b5c2cffa1763.slice/crio-04ccabe77d1fd3b04f613f996eb3b15f8e4d85548d559680fd53392fbf034c7c WatchSource:0}: Error finding container 04ccabe77d1fd3b04f613f996eb3b15f8e4d85548d559680fd53392fbf034c7c: Status 404 returned error can't find the container with id 04ccabe77d1fd3b04f613f996eb3b15f8e4d85548d559680fd53392fbf034c7c Oct 30 13:00:48.056657 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:00:48.056623 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e89920_62df_4684_8d87_a73b607cf8e3.slice/crio-27404ebe284042fb4b9ae0f213d99805aad8709b1d172a0ea025a3eb5d512055 WatchSource:0}: Error finding container 27404ebe284042fb4b9ae0f213d99805aad8709b1d172a0ea025a3eb5d512055: Status 404 returned error can't find the container with id 27404ebe284042fb4b9ae0f213d99805aad8709b1d172a0ea025a3eb5d512055 Oct 30 13:00:48.060670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.060648 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd"] Oct 30 13:00:48.546392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.546239 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" event={"ID":"45e89920-62df-4684-8d87-a73b607cf8e3","Type":"ContainerStarted","Data":"90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b"} Oct 30 13:00:48.546392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.546337 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" event={"ID":"45e89920-62df-4684-8d87-a73b607cf8e3","Type":"ContainerStarted","Data":"27404ebe284042fb4b9ae0f213d99805aad8709b1d172a0ea025a3eb5d512055"} Oct 30 13:00:48.546392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.546367 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:48.548617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.548594 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" event={"ID":"bbb1cb3f-909e-4f38-b293-b5c2cffa1763","Type":"ContainerStarted","Data":"b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be"} Oct 30 13:00:48.548724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.548634 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" event={"ID":"bbb1cb3f-909e-4f38-b293-b5c2cffa1763","Type":"ContainerStarted","Data":"04ccabe77d1fd3b04f613f996eb3b15f8e4d85548d559680fd53392fbf034c7c"} Oct 30 13:00:48.548810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.548792 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:48.552453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.552429 5515 patch_prober.go:29] interesting pod/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:00:48.552560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.552477 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 13:00:48.625775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.625713 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" podStartSLOduration=2.6256932859999997 podStartE2EDuration="2.625693286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:48.589745768 +0000 UTC m=+1666.716546666" watchObservedRunningTime="2026-10-30 13:00:48.625693286 +0000 UTC m=+1666.752494185" Oct 30 13:00:48.634848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.634812 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:48.682249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.682184 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" podStartSLOduration=2.682164854 podStartE2EDuration="2.682164854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:48.670408107 +0000 UTC m=+1666.797209028" watchObservedRunningTime="2026-10-30 13:00:48.682164854 +0000 UTC m=+1666.808965754" Oct 30 13:00:48.762293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.758175 5515 patch_prober.go:29] interesting pod/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:00:48.762293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.758235 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 13:00:48.768996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:48.768966 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:49.552305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:49.552259 5515 patch_prober.go:29] interesting pod/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:00:49.552774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:49.552750 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 13:00:49.749434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:49.749404 5515 patch_prober.go:29] interesting pod/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:00:49.749673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:49.749646 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 13:00:50.384932 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:50.384898 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:50.566486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:50.566177 5515 patch_prober.go:29] interesting pod/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:00:50.566486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:50.566238 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 13:00:50.750074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:50.749982 5515 patch_prober.go:29] interesting pod/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:00:50.750074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:50.750036 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 13:00:51.552318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:51.552262 5515 patch_prober.go:29] interesting pod/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 13:00:51.552502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:51.552337 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 13:00:51.749654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:51.749620 5515 patch_prober.go:29] interesting pod/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:00:51.750068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:51.749671 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 13:00:51.814349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:51.804375 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:51.933003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:51.932948 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:52.557422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:52.557395 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:52.587819 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:52.587795 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:00:52.749126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:52.749098 5515 patch_prober.go:29] interesting pod/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 13:00:52.749310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:52.749152 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 13:00:53.404194 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:53.404159 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:53.753707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:53.753639 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:53.754541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:53.754523 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:00:53.963254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:53.963219 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:00:54.034331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:54.034249 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z"] Oct 30 13:00:54.039250 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:54.039225 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-100c91cd-rtt5z"] Oct 30 13:00:54.900770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:54.899034 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:56.002150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:56.002122 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09e9acd-faec-4cfb-b94c-1e7b04e7fa50" path="/var/lib/kubelet/pods/b09e9acd-faec-4cfb-b94c-1e7b04e7fa50/volumes" Oct 30 13:00:56.486783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:56.486745 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:57.593728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:57.593691 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:57.839639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:57.839604 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:58.117612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:58.103895 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:58.987652 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:00:58.987610 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:00:59.558217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:59.558189 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:00:59.558425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:59.558289 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:00:59.675615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:00:59.675582 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:01.202452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:01.202421 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:01.426503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:01.426465 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:02.900206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:02.900163 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:04.354864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:04.354829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:05.700897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:05.700859 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:05.918522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:05.918486 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:07.468865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:07.468829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:09.116296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:09.102870 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:09.543373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:09.543337 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:10.695734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:10.695695 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:12.132419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:12.132389 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:12.340290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:12.339835 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:13.856334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:13.856286 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:15.396293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:15.392758 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:16.982894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:16.982861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:17.284220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:17.284120 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:01:18.822761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:18.822710 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:20.678292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:20.677799 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:20.992805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:20.992722 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:23.071147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:23.071113 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:24.665030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:24.664978 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:25.845185 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:25.845155 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr"] Oct 30 13:01:25.845698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:25.845560 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" podUID="02b427e7-ed92-46c0-ba0e-e41ce21d2edc" containerName="maas-api" containerID="cri-o://ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0" gracePeriod=30 Oct 30 13:01:26.038973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.038927 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:26.417370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.417338 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 13:01:26.506009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.505962 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:26.521743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.521704 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-openshift-service-ca\") pod \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " Oct 30 13:01:26.522190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.522165 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-metrics-certs\") pod \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " Oct 30 13:01:26.522723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.522700 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-kube-api-access-ms6lb\" (UniqueName: \"kubernetes.io/projected/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-kube-api-access-ms6lb\") pod \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " Oct 30 13:01:26.522905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.522889 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-maas-api-tls\") pod \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\" (UID: \"02b427e7-ed92-46c0-ba0e-e41ce21d2edc\") " Oct 30 13:01:26.532295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.532252 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-openshift-service-ca" pod "02b427e7-ed92-46c0-ba0e-e41ce21d2edc" (UID: "02b427e7-ed92-46c0-ba0e-e41ce21d2edc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:26.536260 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.536220 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-kube-api-access-ms6lb" pod "02b427e7-ed92-46c0-ba0e-e41ce21d2edc" (UID: "02b427e7-ed92-46c0-ba0e-e41ce21d2edc"). InnerVolumeSpecName "kube-api-access-ms6lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:26.548089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.548026 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-metrics-certs" pod "02b427e7-ed92-46c0-ba0e-e41ce21d2edc" (UID: "02b427e7-ed92-46c0-ba0e-e41ce21d2edc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:26.548832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.548767 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-maas-api-tls" pod "02b427e7-ed92-46c0-ba0e-e41ce21d2edc" (UID: "02b427e7-ed92-46c0-ba0e-e41ce21d2edc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:26.625567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.625536 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:26.625567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.625562 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:26.625567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.625571 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms6lb\" (UniqueName: \"kubernetes.io/projected/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-kube-api-access-ms6lb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:26.625797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.625579 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b427e7-ed92-46c0-ba0e-e41ce21d2edc-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:26.755370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.751863 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:26.927588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.927553 5515 generic.go:356] "Generic (PLEG): container finished" podID="02b427e7-ed92-46c0-ba0e-e41ce21d2edc" containerID="ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0" exitCode=0 Oct 30 13:01:26.928024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.927594 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" event={"ID":"02b427e7-ed92-46c0-ba0e-e41ce21d2edc","Type":"ContainerDied","Data":"ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0"} Oct 30 13:01:26.928024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.927647 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" event={"ID":"02b427e7-ed92-46c0-ba0e-e41ce21d2edc","Type":"ContainerDied","Data":"6e378a94af824ac821e6e8db771e44ca7bef186458a6bf5f687445191db4ef75"} Oct 30 13:01:26.928024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.927671 5515 scope.go:122] "RemoveContainer" containerID="ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0" Oct 30 13:01:26.928024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.927602 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" Oct 30 13:01:26.938109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.938090 5515 scope.go:122] "RemoveContainer" containerID="ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0" Oct 30 13:01:26.938417 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:01:26.938387 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0\": container with ID starting with ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0 not found: ID does not exist" containerID="ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0" Oct 30 13:01:26.938502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.938423 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0"} err="failed to get container status \"ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0\": rpc error: code = NotFound desc = could not find container \"ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0\": container with ID starting with ce4226b4be4dfe413b4e44a40fd5cf2c64f68c03d157bcff1729da32c85d3ab0 not found: ID does not exist" Oct 30 13:01:26.982340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.982309 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr"] Oct 30 13:01:26.993675 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:26.993651 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr"] Oct 30 13:01:27.159438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:27.159408 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:28.002521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:28.002489 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b427e7-ed92-46c0-ba0e-e41ce21d2edc" path="/var/lib/kubelet/pods/02b427e7-ed92-46c0-ba0e-e41ce21d2edc/volumes" Oct 30 13:01:28.277341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:28.276296 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:28.988554 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:01:28.988520 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:01:29.044312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:29.044261 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-87cae1-7698fb78c5-6wsbr" podUID="02b427e7-ed92-46c0-ba0e-e41ce21d2edc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.89:8443/health\": context deadline exceeded" Oct 30 13:01:29.872459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:29.871928 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:31.506130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.506096 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:31.560931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.560900 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11526-de39c3"] Oct 30 13:01:31.561666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.561650 5515 cpu_manager.go:414] "containerMap: removing container" podUID="02b427e7-ed92-46c0-ba0e-e41ce21d2edc" containerName="maas-api" Oct 30 13:01:31.561732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.561667 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b427e7-ed92-46c0-ba0e-e41ce21d2edc" containerName="maas-api" Oct 30 13:01:31.561796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.561787 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b427e7-ed92-46c0-ba0e-e41ce21d2edc" containerName="maas-api" Oct 30 13:01:31.565703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.565682 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-de39c3" Oct 30 13:01:31.583624 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.583597 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11526-de39c3"] Oct 30 13:01:31.592056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.592033 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ljkrn\"" Oct 30 13:01:31.665702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.665669 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q7gh\" (UniqueName: \"kubernetes.io/projected/1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6-kube-api-access-2q7gh\") pod \"test-curl-11526-de39c3\" (UID: \"1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6\") " pod="opendatahub/test-curl-11526-de39c3" Oct 30 13:01:31.767033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.766953 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q7gh\" (UniqueName: \"kubernetes.io/projected/1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6-kube-api-access-2q7gh\") pod \"test-curl-11526-de39c3\" (UID: \"1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6\") " pod="opendatahub/test-curl-11526-de39c3" Oct 30 13:01:31.790867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.790836 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q7gh\" (UniqueName: \"kubernetes.io/projected/1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6-kube-api-access-2q7gh\") pod \"test-curl-11526-de39c3\" (UID: \"1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6\") " pod="opendatahub/test-curl-11526-de39c3" Oct 30 13:01:31.876794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:31.876761 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-de39c3" Oct 30 13:01:32.172534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:32.172506 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11526-de39c3"] Oct 30 13:01:32.173136 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:01:32.173113 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2c3db3_6a43_4e9e_9b47_2dd09965a8a6.slice/crio-defd93d37c731f9c304783b7c6045e1570d65c7d7b9d3d2f3f3fb6d5de40009a WatchSource:0}: Error finding container defd93d37c731f9c304783b7c6045e1570d65c7d7b9d3d2f3f3fb6d5de40009a: Status 404 returned error can't find the container with id defd93d37c731f9c304783b7c6045e1570d65c7d7b9d3d2f3f3fb6d5de40009a Oct 30 13:01:32.706687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:32.706634 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:32.951885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:32.950727 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:32.996000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:32.995919 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11526-de39c3" event={"ID":"1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6","Type":"ContainerStarted","Data":"defd93d37c731f9c304783b7c6045e1570d65c7d7b9d3d2f3f3fb6d5de40009a"} Oct 30 13:01:33.169688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:33.169655 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:34.128179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:34.128140 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:34.347204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:34.347167 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:34.364367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:34.364336 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6"] Oct 30 13:01:34.364604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:34.364579 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" containerID="cri-o://ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29" gracePeriod=30 Oct 30 13:01:34.568291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:34.562672 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:34.772830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:34.772792 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:36.252959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:36.252873 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:37.025042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:37.025008 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11526-de39c3" event={"ID":"1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6","Type":"ContainerStarted","Data":"271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b"} Oct 30 13:01:37.067806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:37.067761 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11526-de39c3" podStartSLOduration=1.34285093 podStartE2EDuration="6.067747945s" totalImagesPullingTime="4.724897015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:37.065063463 +0000 UTC m=+1715.191864363" watchObservedRunningTime="2026-10-30 13:01:37.067747945 +0000 UTC m=+1715.194548843" Oct 30 13:01:37.895106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:37.895070 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:38.139419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:38.139386 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11526-de39c3"] Oct 30 13:01:38.149587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:38.149518 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11526-de39c3"] Oct 30 13:01:39.041599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.041561 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11526-de39c3" podUID="1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6" containerName="test-curl-11526-de39c3" containerID="cri-o://271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b" gracePeriod=2 Oct 30 13:01:39.055157 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:01:39.055124 5515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11526-de39c3\" is forbidden: User \"system:node:ip-10-0-134-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-193.ec2.internal' and this object" podUID="1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6" pod="opendatahub/test-curl-11526-de39c3" Oct 30 13:01:39.495089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.493756 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:39.695846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.695823 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 13:01:39.709564 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:01:39.709535 5515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11526-de39c3\" is forbidden: User \"system:node:ip-10-0-134-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-193.ec2.internal' and this object" podUID="1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6" pod="opendatahub/test-curl-11526-de39c3" Oct 30 13:01:39.742744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.742716 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-socket\") pod \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " Oct 30 13:01:39.742861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.742760 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-podinfo\") pod \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " Oct 30 13:01:39.742861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.742794 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-credential-socket\") pod \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " Oct 30 13:01:39.742861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.742835 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-data\") pod \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " Oct 30 13:01:39.742973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.742947 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-envoy\") pod \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " Oct 30 13:01:39.743014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.742965 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-socket" pod "b2d99f7d-5ea2-45ee-b42d-21bd19c07442" (UID: "b2d99f7d-5ea2-45ee-b42d-21bd19c07442"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:39.743068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743053 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istiod-ca-cert\") pod \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " Oct 30 13:01:39.743119 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743104 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-credential-socket" pod "b2d99f7d-5ea2-45ee-b42d-21bd19c07442" (UID: "b2d99f7d-5ea2-45ee-b42d-21bd19c07442"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:39.743172 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743114 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-token\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-token\") pod \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " Oct 30 13:01:39.743172 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743148 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-certs\") pod \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " Oct 30 13:01:39.743318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743159 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-data" pod "b2d99f7d-5ea2-45ee-b42d-21bd19c07442" (UID: "b2d99f7d-5ea2-45ee-b42d-21bd19c07442"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:39.743318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743251 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-kube-api-access-x5bqp\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-kube-api-access-x5bqp\") pod \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\" (UID: \"b2d99f7d-5ea2-45ee-b42d-21bd19c07442\") " Oct 30 13:01:39.743537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743504 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-certs" pod "b2d99f7d-5ea2-45ee-b42d-21bd19c07442" (UID: "b2d99f7d-5ea2-45ee-b42d-21bd19c07442"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:39.743706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743688 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istiod-ca-cert" pod "b2d99f7d-5ea2-45ee-b42d-21bd19c07442" (UID: "b2d99f7d-5ea2-45ee-b42d-21bd19c07442"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:39.743972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743953 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.744046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743977 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.744046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743987 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.744046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.743995 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.744046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.744003 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.745257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.745236 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-kube-api-access-x5bqp" pod "b2d99f7d-5ea2-45ee-b42d-21bd19c07442" (UID: "b2d99f7d-5ea2-45ee-b42d-21bd19c07442"). InnerVolumeSpecName "kube-api-access-x5bqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:39.745257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.745245 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-envoy" pod "b2d99f7d-5ea2-45ee-b42d-21bd19c07442" (UID: "b2d99f7d-5ea2-45ee-b42d-21bd19c07442"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:39.745437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.745391 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-podinfo" pod "b2d99f7d-5ea2-45ee-b42d-21bd19c07442" (UID: "b2d99f7d-5ea2-45ee-b42d-21bd19c07442"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:01:39.745519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.745496 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-token" pod "b2d99f7d-5ea2-45ee-b42d-21bd19c07442" (UID: "b2d99f7d-5ea2-45ee-b42d-21bd19c07442"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:39.845236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.845205 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.845236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.845230 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.845236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.845239 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.845465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:39.845247 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5bqp\" (UniqueName: \"kubernetes.io/projected/b2d99f7d-5ea2-45ee-b42d-21bd19c07442-kube-api-access-x5bqp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:40.049746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.049717 5515 generic.go:356] "Generic (PLEG): container finished" podID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerID="ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29" exitCode=0 Oct 30 13:01:40.050179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.049764 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" Oct 30 13:01:40.050179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.049800 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" event={"ID":"b2d99f7d-5ea2-45ee-b42d-21bd19c07442","Type":"ContainerDied","Data":"ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29"} Oct 30 13:01:40.050179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.049856 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6" event={"ID":"b2d99f7d-5ea2-45ee-b42d-21bd19c07442","Type":"ContainerDied","Data":"fa01f8f8bac8bd0b8adf0ddef04ac8303f591ab33710b0fa3c9f10d948b50792"} Oct 30 13:01:40.050179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.049890 5515 scope.go:122] "RemoveContainer" containerID="ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29" Oct 30 13:01:40.059695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.059662 5515 scope.go:122] "RemoveContainer" containerID="e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc" Oct 30 13:01:40.069208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.069193 5515 scope.go:122] "RemoveContainer" containerID="ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29" Oct 30 13:01:40.069452 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:01:40.069436 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29\": container with ID starting with ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29 not found: ID does not exist" containerID="ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29" Oct 30 13:01:40.069517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.069459 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29"} err="failed to get container status \"ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29\": rpc error: code = NotFound desc = could not find container \"ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29\": container with ID starting with ae0371b4f0c922152f363b723999225fc36bb722424642d0332d759bd5645d29 not found: ID does not exist" Oct 30 13:01:40.069517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.069474 5515 scope.go:122] "RemoveContainer" containerID="e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc" Oct 30 13:01:40.069699 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:01:40.069684 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc\": container with ID starting with e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc not found: ID does not exist" containerID="e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc" Oct 30 13:01:40.069737 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.069703 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc"} err="failed to get container status \"e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc\": rpc error: code = NotFound desc = could not find container \"e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc\": container with ID starting with e8c720652b0582ce6789274a8b4453de6b21d7d3f3dd017ab85d68ed9c7beedc not found: ID does not exist" Oct 30 13:01:40.103588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.103563 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6"] Oct 30 13:01:40.115226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.115200 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-87cae1-openshift-default-55bbb4f6cf-j9kc6"] Oct 30 13:01:40.983867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:40.983831 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:41.265536 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:41.265513 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-de39c3" Oct 30 13:01:41.365526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:41.365499 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6-kube-api-access-2q7gh\" (UniqueName: \"kubernetes.io/projected/1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6-kube-api-access-2q7gh\") pod \"1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6\" (UID: \"1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6\") " Oct 30 13:01:41.367489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:41.367462 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6-kube-api-access-2q7gh" pod "1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6" (UID: "1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6"). InnerVolumeSpecName "kube-api-access-2q7gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:41.467612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:41.467584 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q7gh\" (UniqueName: \"kubernetes.io/projected/1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6-kube-api-access-2q7gh\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:01:42.002688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:42.002618 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6" path="/var/lib/kubelet/pods/1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6/volumes" Oct 30 13:01:42.003097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:42.003083 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" path="/var/lib/kubelet/pods/b2d99f7d-5ea2-45ee-b42d-21bd19c07442/volumes" Oct 30 13:01:42.067126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:42.067101 5515 generic.go:356] "Generic (PLEG): container finished" podID="1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6" containerID="271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b" exitCode=137 Oct 30 13:01:42.067252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:42.067140 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-de39c3" Oct 30 13:01:42.067252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:42.067192 5515 scope.go:122] "RemoveContainer" containerID="271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b" Oct 30 13:01:42.076200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:42.076185 5515 scope.go:122] "RemoveContainer" containerID="271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b" Oct 30 13:01:42.076485 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:01:42.076465 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b\": container with ID starting with 271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b not found: ID does not exist" containerID="271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b" Oct 30 13:01:42.076573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:42.076491 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b"} err="failed to get container status \"271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b\": rpc error: code = NotFound desc = could not find container \"271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b\": container with ID starting with 271f9e9dce5d637f6b5540e3934116efedee120c31d56b1dea7793355b46d63b not found: ID does not exist" Oct 30 13:01:42.623614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:42.622969 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:44.161754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:44.161713 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:45.677845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:45.677809 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:45.912594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:45.912562 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:47.305291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:47.301861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:48.771815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:48.771781 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:49.262532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:49.262501 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv"] Oct 30 13:01:49.301111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:49.301080 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z"] Oct 30 13:01:49.530920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:49.530830 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:49.741246 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:49.741204 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:50.438906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:50.438873 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:51.985144 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:51.985040 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:53.593760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:53.593716 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:54.262891 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.262852 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" podUID="8ba1def9-8934-459f-b57d-908c661162f9" containerName="payload-processing" containerID="cri-o://44e7d370c9686611ab9ef60488dafea69690f0456607ef454543e13bc8c8aaf7" gracePeriod=25 Oct 30 13:01:54.268066 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.268031 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:54.302208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.302178 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" podUID="01d07376-a4be-404d-8677-16556eae482a" containerName="payload-pre-processing" containerID="cri-o://b53ddf4e9dd54f7ebec99603483395b805e2cbe0ecc44d3d688a34c248816a38" gracePeriod=25 Oct 30 13:01:54.622746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.622672 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd"] Oct 30 13:01:54.623689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.623664 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" Oct 30 13:01:54.623781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.623692 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" Oct 30 13:01:54.623781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.623710 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" Oct 30 13:01:54.623781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.623719 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" Oct 30 13:01:54.623781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.623760 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6" containerName="test-curl-11526-de39c3" Oct 30 13:01:54.623781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.623767 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6" containerName="test-curl-11526-de39c3" Oct 30 13:01:54.623987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.623973 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" Oct 30 13:01:54.624021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.623995 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d99f7d-5ea2-45ee-b42d-21bd19c07442" containerName="istio-proxy" Oct 30 13:01:54.624021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.624012 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2c3db3-6a43-4e9e-9b47-2dd09965a8a6" containerName="test-curl-11526-de39c3" Oct 30 13:01:54.627327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.627304 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.642922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.642901 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f60f16-openshift-default-dockercfg-bd8r6\"" Oct 30 13:01:54.657683 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.657655 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd"] Oct 30 13:01:54.695140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.695104 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-socket\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.695358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.695338 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-credential-socket\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.695414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.695377 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-podinfo\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.695528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.695510 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-data\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.695596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.695543 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmtls\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-kube-api-access-lmtls\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.695667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.695652 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-certs\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.695724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.695693 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-envoy\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.695799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.695786 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-token\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.695875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.695861 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istiod-ca-cert\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.765466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.765133 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.809080 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-credential-socket\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.809131 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-podinfo\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.809202 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-data\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.809227 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmtls\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-kube-api-access-lmtls\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.809307 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-certs\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.809344 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-envoy\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.809400 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-token\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.809466 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istiod-ca-cert\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.809535 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-socket\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.810294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.810220 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-socket\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.811967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.811858 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istiod-ca-cert\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.816289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.812370 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-certs\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.816289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.812604 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-data\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.816790 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.816768 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-credential-socket\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.843726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.820003 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-podinfo\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.865257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.865224 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmtls\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-kube-api-access-lmtls\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.865257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.865706 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-envoy\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.882812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.882783 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-token\") pod \"e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:54.946291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:54.943239 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:55.012497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:55.011642 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:55.263676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:55.256391 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:55.369219 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:01:55.369187 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0cbb8a_8a4c_440a_923b_3aa8a52ac251.slice/crio-288c81e7087d29b537a58078d9a4aca7b0aaa16823f59d02a7a3a8227c32506b WatchSource:0}: Error finding container 288c81e7087d29b537a58078d9a4aca7b0aaa16823f59d02a7a3a8227c32506b: Status 404 returned error can't find the container with id 288c81e7087d29b537a58078d9a4aca7b0aaa16823f59d02a7a3a8227c32506b Oct 30 13:01:55.370151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:55.370125 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd"] Oct 30 13:01:55.371741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:55.371702 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:01:55.371844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:55.371780 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:01:55.371844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:55.371823 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:01:56.269096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:56.269012 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" event={"ID":"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251","Type":"ContainerStarted","Data":"da085127e1616356c1e3f0a0c066b661d7f13ecc8b23383464f7439601af1938"} Oct 30 13:01:56.269096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:56.269066 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" event={"ID":"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251","Type":"ContainerStarted","Data":"288c81e7087d29b537a58078d9a4aca7b0aaa16823f59d02a7a3a8227c32506b"} Oct 30 13:01:56.328211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:56.328153 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" podStartSLOduration=2.328139186 podStartE2EDuration="2.328139186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:56.325201109 +0000 UTC m=+1734.452002008" watchObservedRunningTime="2026-10-30 13:01:56.328139186 +0000 UTC m=+1734.454940084" Oct 30 13:01:56.556334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:56.556241 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:56.956945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:56.956914 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:56.987847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:56.986937 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:57.065058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:57.064990 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:57.279621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:57.279535 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:57.280395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:57.280376 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:01:57.315783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:57.315750 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:57.755887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:57.755856 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:58.061748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:58.061665 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" podUID="8ba1def9-8934-459f-b57d-908c661162f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:01:58.061748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:58.061717 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" podUID="01d07376-a4be-404d-8677-16556eae482a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:01:58.974091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:58.974061 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:58.988138 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:01:58.988108 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:01:59.124293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:59.118678 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:01:59.212294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:01:59.203801 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:00.823947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:00.823908 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:01.110346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:01.101714 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:02.298770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:02.298739 5515 scope.go:122] "RemoveContainer" containerID="56a8c55b70c2b3f7ab9b9e8a17bc7c37c407812f15497d28b0546098ccfab8b8" Oct 30 13:02:02.482444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:02.482407 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:04.136590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:04.136562 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:04.691241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:04.685321 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:04.903899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:04.903852 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5"] Oct 30 13:02:04.904117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:04.904090 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" containerID="cri-o://054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72" gracePeriod=30 Oct 30 13:02:04.913771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:04.913701 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:05.202163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:05.195978 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:05.749761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:05.749722 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:07.204026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:07.203984 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:07.715241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:07.715155 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:07.949553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:07.949498 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:08.143307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:08.140543 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" podUID="8ba1def9-8934-459f-b57d-908c661162f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:02:08.159455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:08.154417 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" podUID="01d07376-a4be-404d-8677-16556eae482a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:02:08.756793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:08.756758 5515 patch_prober.go:29] interesting pod/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:02:08.757174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:08.756809 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:02:09.290216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:09.290184 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:09.826632 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:09.826595 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:10.133088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.133050 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:10.178165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.178142 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:02:10.299801 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.299759 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-podinfo\") pod \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " Oct 30 13:02:10.300000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.299879 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-credential-socket\") pod \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " Oct 30 13:02:10.300000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.299919 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-data\") pod \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " Oct 30 13:02:10.300000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.299983 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-certs\") pod \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " Oct 30 13:02:10.300189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300072 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-socket\") pod \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " Oct 30 13:02:10.300189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300133 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istiod-ca-cert\") pod \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " Oct 30 13:02:10.300297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300203 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-credential-socket" pod "affad291-ae66-47ba-9b0f-f1aeb0ff424e" (UID: "affad291-ae66-47ba-9b0f-f1aeb0ff424e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:10.300297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300229 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-kube-api-access-bdczg\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-kube-api-access-bdczg\") pod \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " Oct 30 13:02:10.300386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300323 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-data" pod "affad291-ae66-47ba-9b0f-f1aeb0ff424e" (UID: "affad291-ae66-47ba-9b0f-f1aeb0ff424e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:10.300386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300347 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-socket" pod "affad291-ae66-47ba-9b0f-f1aeb0ff424e" (UID: "affad291-ae66-47ba-9b0f-f1aeb0ff424e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:10.300386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300338 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-certs" pod "affad291-ae66-47ba-9b0f-f1aeb0ff424e" (UID: "affad291-ae66-47ba-9b0f-f1aeb0ff424e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:10.300386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300359 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-envoy\") pod \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " Oct 30 13:02:10.300543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300392 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-token\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-token\") pod \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\" (UID: \"affad291-ae66-47ba-9b0f-f1aeb0ff424e\") " Oct 30 13:02:10.300584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300553 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istiod-ca-cert" pod "affad291-ae66-47ba-9b0f-f1aeb0ff424e" (UID: "affad291-ae66-47ba-9b0f-f1aeb0ff424e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:10.300951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300921 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:10.300951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300947 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:10.300951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300961 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:10.301156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300973 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:10.301156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.300984 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:10.302427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.302401 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-podinfo" pod "affad291-ae66-47ba-9b0f-f1aeb0ff424e" (UID: "affad291-ae66-47ba-9b0f-f1aeb0ff424e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:10.302529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.302447 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-envoy" pod "affad291-ae66-47ba-9b0f-f1aeb0ff424e" (UID: "affad291-ae66-47ba-9b0f-f1aeb0ff424e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:10.302723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.302703 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-kube-api-access-bdczg" pod "affad291-ae66-47ba-9b0f-f1aeb0ff424e" (UID: "affad291-ae66-47ba-9b0f-f1aeb0ff424e"). InnerVolumeSpecName "kube-api-access-bdczg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:10.302766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.302733 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-token" pod "affad291-ae66-47ba-9b0f-f1aeb0ff424e" (UID: "affad291-ae66-47ba-9b0f-f1aeb0ff424e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:10.402232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.402202 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdczg\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-kube-api-access-bdczg\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:10.402232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.402227 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:10.402232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.402237 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:10.402457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.402245 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affad291-ae66-47ba-9b0f-f1aeb0ff424e-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:10.468704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.468677 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:10.494714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.494688 5515 generic.go:356] "Generic (PLEG): container finished" podID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerID="054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72" exitCode=0 Oct 30 13:02:10.494843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.494743 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" Oct 30 13:02:10.494843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.494775 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" event={"ID":"affad291-ae66-47ba-9b0f-f1aeb0ff424e","Type":"ContainerDied","Data":"054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72"} Oct 30 13:02:10.494843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.494814 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5" event={"ID":"affad291-ae66-47ba-9b0f-f1aeb0ff424e","Type":"ContainerDied","Data":"ce2814572afc3c7938cda903d574059e4ae203c18da23caf3048bb4ee52aea25"} Oct 30 13:02:10.494843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.494836 5515 scope.go:122] "RemoveContainer" containerID="054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72" Oct 30 13:02:10.505745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.505726 5515 scope.go:122] "RemoveContainer" containerID="fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4" Oct 30 13:02:10.514589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.514565 5515 scope.go:122] "RemoveContainer" containerID="054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72" Oct 30 13:02:10.514845 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:02:10.514826 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72\": container with ID starting with 054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72 not found: ID does not exist" containerID="054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72" Oct 30 13:02:10.514900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.514853 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72"} err="failed to get container status \"054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72\": rpc error: code = NotFound desc = could not find container \"054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72\": container with ID starting with 054ec03dee12041a7285daeac1f1fe59ee28c704c545cedb34c3c932219e6d72 not found: ID does not exist" Oct 30 13:02:10.514900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.514870 5515 scope.go:122] "RemoveContainer" containerID="fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4" Oct 30 13:02:10.515128 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:02:10.515110 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4\": container with ID starting with fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4 not found: ID does not exist" containerID="fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4" Oct 30 13:02:10.515169 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.515134 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4"} err="failed to get container status \"fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4\": rpc error: code = NotFound desc = could not find container \"fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4\": container with ID starting with fe33681ca0669ae3a7df75fa1f088cc3057d2848d63bb852437c06b388259ac4 not found: ID does not exist" Oct 30 13:02:10.547635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.547605 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5"] Oct 30 13:02:10.559727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:10.559699 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2e0e28-openshift-default-cf88cc7c7-mjsl5"] Oct 30 13:02:11.666291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:11.660767 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:11.895082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:11.895051 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:12.004366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:12.004292 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" path="/var/lib/kubelet/pods/affad291-ae66-47ba-9b0f-f1aeb0ff424e/volumes" Oct 30 13:02:12.178459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:12.178428 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:13.517980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:13.517934 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:14.535609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.535408 5515 generic.go:356] "Generic (PLEG): container finished" podID="8ba1def9-8934-459f-b57d-908c661162f9" containerID="44e7d370c9686611ab9ef60488dafea69690f0456607ef454543e13bc8c8aaf7" exitCode=0 Oct 30 13:02:14.537327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.535475 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" event={"ID":"8ba1def9-8934-459f-b57d-908c661162f9","Type":"ContainerDied","Data":"44e7d370c9686611ab9ef60488dafea69690f0456607ef454543e13bc8c8aaf7"} Oct 30 13:02:14.538500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.538472 5515 generic.go:356] "Generic (PLEG): container finished" podID="01d07376-a4be-404d-8677-16556eae482a" containerID="b53ddf4e9dd54f7ebec99603483395b805e2cbe0ecc44d3d688a34c248816a38" exitCode=0 Oct 30 13:02:14.538689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.538667 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" event={"ID":"01d07376-a4be-404d-8677-16556eae482a","Type":"ContainerDied","Data":"b53ddf4e9dd54f7ebec99603483395b805e2cbe0ecc44d3d688a34c248816a38"} Oct 30 13:02:14.574917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.574893 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 13:02:14.594628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.594608 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 13:02:14.678385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.678360 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d07376-a4be-404d-8677-16556eae482a-kube-api-access-v7bbw\" (UniqueName: \"kubernetes.io/projected/01d07376-a4be-404d-8677-16556eae482a-kube-api-access-v7bbw\") pod \"01d07376-a4be-404d-8677-16556eae482a\" (UID: \"01d07376-a4be-404d-8677-16556eae482a\") " Oct 30 13:02:14.678546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.678475 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d07376-a4be-404d-8677-16556eae482a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d07376-a4be-404d-8677-16556eae482a-plugins-config-volume\") pod \"01d07376-a4be-404d-8677-16556eae482a\" (UID: \"01d07376-a4be-404d-8677-16556eae482a\") " Oct 30 13:02:14.678679 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.678664 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba1def9-8934-459f-b57d-908c661162f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1def9-8934-459f-b57d-908c661162f9-plugins-config-volume\") pod \"8ba1def9-8934-459f-b57d-908c661162f9\" (UID: \"8ba1def9-8934-459f-b57d-908c661162f9\") " Oct 30 13:02:14.678741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.678731 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba1def9-8934-459f-b57d-908c661162f9-kube-api-access-z957d\" (UniqueName: \"kubernetes.io/projected/8ba1def9-8934-459f-b57d-908c661162f9-kube-api-access-z957d\") pod \"8ba1def9-8934-459f-b57d-908c661162f9\" (UID: \"8ba1def9-8934-459f-b57d-908c661162f9\") " Oct 30 13:02:14.679037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.678897 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d07376-a4be-404d-8677-16556eae482a-plugins-config-volume" pod "01d07376-a4be-404d-8677-16556eae482a" (UID: "01d07376-a4be-404d-8677-16556eae482a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:14.679138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.679107 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba1def9-8934-459f-b57d-908c661162f9-plugins-config-volume" pod "8ba1def9-8934-459f-b57d-908c661162f9" (UID: "8ba1def9-8934-459f-b57d-908c661162f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:14.679342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.679319 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d07376-a4be-404d-8677-16556eae482a-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:14.679424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.679345 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba1def9-8934-459f-b57d-908c661162f9-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:14.681111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.681090 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d07376-a4be-404d-8677-16556eae482a-kube-api-access-v7bbw" pod "01d07376-a4be-404d-8677-16556eae482a" (UID: "01d07376-a4be-404d-8677-16556eae482a"). InnerVolumeSpecName "kube-api-access-v7bbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:14.682079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.682056 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba1def9-8934-459f-b57d-908c661162f9-kube-api-access-z957d" pod "8ba1def9-8934-459f-b57d-908c661162f9" (UID: "8ba1def9-8934-459f-b57d-908c661162f9"). InnerVolumeSpecName "kube-api-access-z957d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:14.780718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.780688 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z957d\" (UniqueName: \"kubernetes.io/projected/8ba1def9-8934-459f-b57d-908c661162f9-kube-api-access-z957d\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:14.780718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:14.780716 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7bbw\" (UniqueName: \"kubernetes.io/projected/01d07376-a4be-404d-8677-16556eae482a-kube-api-access-v7bbw\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:15.254913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.254876 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:15.546044 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.545967 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" Oct 30 13:02:15.546495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.545967 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv" event={"ID":"8ba1def9-8934-459f-b57d-908c661162f9","Type":"ContainerDied","Data":"851be20b5265e9d91221074c89609cc61a95391044adc292b29d0fa7819fbb6e"} Oct 30 13:02:15.546495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.546104 5515 scope.go:122] "RemoveContainer" containerID="44e7d370c9686611ab9ef60488dafea69690f0456607ef454543e13bc8c8aaf7" Oct 30 13:02:15.547620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.547496 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" Oct 30 13:02:15.547620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.547523 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z" event={"ID":"01d07376-a4be-404d-8677-16556eae482a","Type":"ContainerDied","Data":"7ace092342f6511c0e211b82c1502deef6babf79a78fe5c5e86b71cbf9722b03"} Oct 30 13:02:15.559115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.559099 5515 scope.go:122] "RemoveContainer" containerID="b53ddf4e9dd54f7ebec99603483395b805e2cbe0ecc44d3d688a34c248816a38" Oct 30 13:02:15.599318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.599289 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z"] Oct 30 13:02:15.610500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.610473 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-tzp9z"] Oct 30 13:02:15.640894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.640868 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv"] Oct 30 13:02:15.649540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:15.649516 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-xspsv"] Oct 30 13:02:16.002901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:16.002871 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d07376-a4be-404d-8677-16556eae482a" path="/var/lib/kubelet/pods/01d07376-a4be-404d-8677-16556eae482a/volumes" Oct 30 13:02:16.003328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:16.003314 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba1def9-8934-459f-b57d-908c661162f9" path="/var/lib/kubelet/pods/8ba1def9-8934-459f-b57d-908c661162f9/volumes" Oct 30 13:02:16.022173 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:16.022145 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd"] Oct 30 13:02:16.060115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:16.060088 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk"] Oct 30 13:02:16.321405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:16.320106 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:16.565152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:16.565120 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:16.843581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:16.843546 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:18.360880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:18.360837 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:18.623532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:18.623439 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:18.689737 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:18.689702 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4"] Oct 30 13:02:18.690235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:18.690199 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" containerID="cri-o://0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a" gracePeriod=30 Oct 30 13:02:18.975311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:18.973749 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:19.682946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:19.682907 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:19.981205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:19.981109 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:20.380690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.380603 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:20.593490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.593458 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r"] Oct 30 13:02:20.594480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594454 5515 cpu_manager.go:414] "containerMap: removing container" podUID="8ba1def9-8934-459f-b57d-908c661162f9" containerName="payload-processing" Oct 30 13:02:20.594480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594479 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba1def9-8934-459f-b57d-908c661162f9" containerName="payload-processing" Oct 30 13:02:20.594480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594494 5515 cpu_manager.go:414] "containerMap: removing container" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" Oct 30 13:02:20.594750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594504 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" Oct 30 13:02:20.594750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594518 5515 cpu_manager.go:414] "containerMap: removing container" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" Oct 30 13:02:20.594750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594525 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" Oct 30 13:02:20.594750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594575 5515 cpu_manager.go:414] "containerMap: removing container" podUID="01d07376-a4be-404d-8677-16556eae482a" containerName="payload-pre-processing" Oct 30 13:02:20.594750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594582 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d07376-a4be-404d-8677-16556eae482a" containerName="payload-pre-processing" Oct 30 13:02:20.594750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594737 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" Oct 30 13:02:20.594750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594753 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba1def9-8934-459f-b57d-908c661162f9" containerName="payload-processing" Oct 30 13:02:20.595070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594764 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="affad291-ae66-47ba-9b0f-f1aeb0ff424e" containerName="istio-proxy" Oct 30 13:02:20.595070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.594783 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d07376-a4be-404d-8677-16556eae482a" containerName="payload-pre-processing" Oct 30 13:02:20.598823 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.598797 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:20.618546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.618513 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r"] Oct 30 13:02:20.624729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.624699 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ea117f29\"" Oct 30 13:02:20.624917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.624804 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ea117f29-dockercfg-m55dp\"" Oct 30 13:02:20.636901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.636865 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:20.736648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.735729 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d303af-cdff-4416-bdad-5f493d7c1343-plugins-config-volume\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r\" (UID: \"f3d303af-cdff-4416-bdad-5f493d7c1343\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:20.736648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.735807 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9q6c\" (UniqueName: \"kubernetes.io/projected/f3d303af-cdff-4416-bdad-5f493d7c1343-kube-api-access-w9q6c\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r\" (UID: \"f3d303af-cdff-4416-bdad-5f493d7c1343\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:20.874242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.874190 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d303af-cdff-4416-bdad-5f493d7c1343-plugins-config-volume\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r\" (UID: \"f3d303af-cdff-4416-bdad-5f493d7c1343\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:20.875743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.875720 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9q6c\" (UniqueName: \"kubernetes.io/projected/f3d303af-cdff-4416-bdad-5f493d7c1343-kube-api-access-w9q6c\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r\" (UID: \"f3d303af-cdff-4416-bdad-5f493d7c1343\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:20.876984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.876946 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d303af-cdff-4416-bdad-5f493d7c1343-plugins-config-volume\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r\" (UID: \"f3d303af-cdff-4416-bdad-5f493d7c1343\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:20.909986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:20.909903 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:21.011013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.010978 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9q6c\" (UniqueName: \"kubernetes.io/projected/f3d303af-cdff-4416-bdad-5f493d7c1343-kube-api-access-w9q6c\") pod \"payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r\" (UID: \"f3d303af-cdff-4416-bdad-5f493d7c1343\") " pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:21.026339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.023206 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" podUID="45e89920-62df-4684-8d87-a73b607cf8e3" containerName="payload-processing" containerID="cri-o://90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b" gracePeriod=25 Oct 30 13:02:21.061371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.061317 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" podUID="bbb1cb3f-909e-4f38-b293-b5c2cffa1763" containerName="payload-pre-processing" containerID="cri-o://b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be" gracePeriod=25 Oct 30 13:02:21.214502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.214466 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:21.326895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.326496 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:02:21.399440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.399412 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:02:21.491760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.491673 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e89920-62df-4684-8d87-a73b607cf8e3-kube-api-access-nwv2s\" (UniqueName: \"kubernetes.io/projected/45e89920-62df-4684-8d87-a73b607cf8e3-kube-api-access-nwv2s\") pod \"45e89920-62df-4684-8d87-a73b607cf8e3\" (UID: \"45e89920-62df-4684-8d87-a73b607cf8e3\") " Oct 30 13:02:21.491932 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.491881 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-plugins-config-volume\") pod \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\" (UID: \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\") " Oct 30 13:02:21.492008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.491969 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e89920-62df-4684-8d87-a73b607cf8e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e89920-62df-4684-8d87-a73b607cf8e3-plugins-config-volume\") pod \"45e89920-62df-4684-8d87-a73b607cf8e3\" (UID: \"45e89920-62df-4684-8d87-a73b607cf8e3\") " Oct 30 13:02:21.492008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.492002 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-kube-api-access-cm664\" (UniqueName: \"kubernetes.io/projected/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-kube-api-access-cm664\") pod \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\" (UID: \"bbb1cb3f-909e-4f38-b293-b5c2cffa1763\") " Oct 30 13:02:21.492522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.492488 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e89920-62df-4684-8d87-a73b607cf8e3-plugins-config-volume" pod "45e89920-62df-4684-8d87-a73b607cf8e3" (UID: "45e89920-62df-4684-8d87-a73b607cf8e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:21.492785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.492762 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e89920-62df-4684-8d87-a73b607cf8e3-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:21.492867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.492759 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-plugins-config-volume" pod "bbb1cb3f-909e-4f38-b293-b5c2cffa1763" (UID: "bbb1cb3f-909e-4f38-b293-b5c2cffa1763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:21.494256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.494226 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e89920-62df-4684-8d87-a73b607cf8e3-kube-api-access-nwv2s" pod "45e89920-62df-4684-8d87-a73b607cf8e3" (UID: "45e89920-62df-4684-8d87-a73b607cf8e3"). InnerVolumeSpecName "kube-api-access-nwv2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:21.495251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.495229 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-kube-api-access-cm664" pod "bbb1cb3f-909e-4f38-b293-b5c2cffa1763" (UID: "bbb1cb3f-909e-4f38-b293-b5c2cffa1763"). InnerVolumeSpecName "kube-api-access-cm664". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:21.577901 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:02:21.577873 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d303af_cdff_4416_bdad_5f493d7c1343.slice/crio-a3cbfcec90ab03f4320c37c9c3325ea720feb70e93f92f22a2bb46fffa0985df WatchSource:0}: Error finding container a3cbfcec90ab03f4320c37c9c3325ea720feb70e93f92f22a2bb46fffa0985df: Status 404 returned error can't find the container with id a3cbfcec90ab03f4320c37c9c3325ea720feb70e93f92f22a2bb46fffa0985df Oct 30 13:02:21.579110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.579078 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r"] Oct 30 13:02:21.593993 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.593966 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:21.594104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.593996 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm664\" (UniqueName: \"kubernetes.io/projected/bbb1cb3f-909e-4f38-b293-b5c2cffa1763-kube-api-access-cm664\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:21.594104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.594011 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwv2s\" (UniqueName: \"kubernetes.io/projected/45e89920-62df-4684-8d87-a73b607cf8e3-kube-api-access-nwv2s\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:21.596539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.596505 5515 generic.go:356] "Generic (PLEG): container finished" podID="45e89920-62df-4684-8d87-a73b607cf8e3" containerID="90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b" exitCode=0 Oct 30 13:02:21.596641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.596550 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" Oct 30 13:02:21.596641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.596548 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" event={"ID":"45e89920-62df-4684-8d87-a73b607cf8e3","Type":"ContainerDied","Data":"90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b"} Oct 30 13:02:21.596641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.596617 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd" event={"ID":"45e89920-62df-4684-8d87-a73b607cf8e3","Type":"ContainerDied","Data":"27404ebe284042fb4b9ae0f213d99805aad8709b1d172a0ea025a3eb5d512055"} Oct 30 13:02:21.596784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.596654 5515 scope.go:122] "RemoveContainer" containerID="90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b" Oct 30 13:02:21.598049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.598026 5515 generic.go:356] "Generic (PLEG): container finished" podID="bbb1cb3f-909e-4f38-b293-b5c2cffa1763" containerID="b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be" exitCode=0 Oct 30 13:02:21.598165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.598050 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" event={"ID":"bbb1cb3f-909e-4f38-b293-b5c2cffa1763","Type":"ContainerDied","Data":"b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be"} Oct 30 13:02:21.598165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.598071 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" Oct 30 13:02:21.598165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.598098 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk" event={"ID":"bbb1cb3f-909e-4f38-b293-b5c2cffa1763","Type":"ContainerDied","Data":"04ccabe77d1fd3b04f613f996eb3b15f8e4d85548d559680fd53392fbf034c7c"} Oct 30 13:02:21.599997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.599969 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" event={"ID":"f3d303af-cdff-4416-bdad-5f493d7c1343","Type":"ContainerStarted","Data":"a3cbfcec90ab03f4320c37c9c3325ea720feb70e93f92f22a2bb46fffa0985df"} Oct 30 13:02:21.622067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.622048 5515 scope.go:122] "RemoveContainer" containerID="90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b" Oct 30 13:02:21.622444 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:02:21.622400 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b\": container with ID starting with 90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b not found: ID does not exist" containerID="90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b" Oct 30 13:02:21.622444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.622432 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b"} err="failed to get container status \"90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b\": rpc error: code = NotFound desc = could not find container \"90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b\": container with ID starting with 90915fce94a680a3b775dc2b4d79b8a2edbb74eeb0057a26a45538a7875fa65b not found: ID does not exist" Oct 30 13:02:21.622590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.622450 5515 scope.go:122] "RemoveContainer" containerID="b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be" Oct 30 13:02:21.637152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.637131 5515 scope.go:122] "RemoveContainer" containerID="b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be" Oct 30 13:02:21.637494 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:02:21.637470 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be\": container with ID starting with b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be not found: ID does not exist" containerID="b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be" Oct 30 13:02:21.637576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.637503 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be"} err="failed to get container status \"b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be\": rpc error: code = NotFound desc = could not find container \"b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be\": container with ID starting with b1d450dffbac2fe70114a1381371cffa2f9e68e570193b9711ae55fcb90035be not found: ID does not exist" Oct 30 13:02:21.678546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.678520 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd"] Oct 30 13:02:21.681239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.681217 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4561d2-6dc4f865fb-rmvnd"] Oct 30 13:02:21.693752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.693728 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd"] Oct 30 13:02:21.694413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.694399 5515 cpu_manager.go:414] "containerMap: removing container" podUID="45e89920-62df-4684-8d87-a73b607cf8e3" containerName="payload-processing" Oct 30 13:02:21.694479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.694414 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e89920-62df-4684-8d87-a73b607cf8e3" containerName="payload-processing" Oct 30 13:02:21.694479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.694436 5515 cpu_manager.go:414] "containerMap: removing container" podUID="bbb1cb3f-909e-4f38-b293-b5c2cffa1763" containerName="payload-pre-processing" Oct 30 13:02:21.694479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.694441 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb1cb3f-909e-4f38-b293-b5c2cffa1763" containerName="payload-pre-processing" Oct 30 13:02:21.694596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.694569 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e89920-62df-4684-8d87-a73b607cf8e3" containerName="payload-processing" Oct 30 13:02:21.694596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.694589 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb1cb3f-909e-4f38-b293-b5c2cffa1763" containerName="payload-pre-processing" Oct 30 13:02:21.697210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.697194 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.715245 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.715215 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ea117f29-serving-cert\"" Oct 30 13:02:21.718565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.718540 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk"] Oct 30 13:02:21.731325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.731261 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd"] Oct 30 13:02:21.735778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.735755 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4561d2-56796f756c-xgkvk"] Oct 30 13:02:21.796136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.796054 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-metrics-certs\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.796136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.796119 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mckbt\" (UniqueName: \"kubernetes.io/projected/fb37b59a-0c72-4523-886e-bd89e1412c14-kube-api-access-mckbt\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.796591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.796220 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-maas-api-tls\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.796591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.796408 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb37b59a-0c72-4523-886e-bd89e1412c14-openshift-service-ca\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.897948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.897915 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb37b59a-0c72-4523-886e-bd89e1412c14-openshift-service-ca\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.898092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.898073 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-metrics-certs\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.898145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.898119 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mckbt\" (UniqueName: \"kubernetes.io/projected/fb37b59a-0c72-4523-886e-bd89e1412c14-kube-api-access-mckbt\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.898196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.898168 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-maas-api-tls\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.898769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.898743 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb37b59a-0c72-4523-886e-bd89e1412c14-openshift-service-ca\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.901240 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.901217 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-maas-api-tls\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.901347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.901224 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-metrics-certs\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.921764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.921740 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mckbt\" (UniqueName: \"kubernetes.io/projected/fb37b59a-0c72-4523-886e-bd89e1412c14-kube-api-access-mckbt\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:21.969563 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:21.969467 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:22.027129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.026151 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:22.055247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.055165 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e89920-62df-4684-8d87-a73b607cf8e3" path="/var/lib/kubelet/pods/45e89920-62df-4684-8d87-a73b607cf8e3/volumes" Oct 30 13:02:22.071930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.071883 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb1cb3f-909e-4f38-b293-b5c2cffa1763" path="/var/lib/kubelet/pods/bbb1cb3f-909e-4f38-b293-b5c2cffa1763/volumes" Oct 30 13:02:22.207918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.207862 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:22.404789 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:02:22.404763 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb37b59a_0c72_4523_886e_bd89e1412c14.slice/crio-8158a2233333e8f00ddaaba2d538eb343e63ef48fd4b341929dc799569cbbb42 WatchSource:0}: Error finding container 8158a2233333e8f00ddaaba2d538eb343e63ef48fd4b341929dc799569cbbb42: Status 404 returned error can't find the container with id 8158a2233333e8f00ddaaba2d538eb343e63ef48fd4b341929dc799569cbbb42 Oct 30 13:02:22.405868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.405841 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd"] Oct 30 13:02:22.589263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.589229 5515 patch_prober.go:29] interesting pod/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:02:22.589457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.589312 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:02:22.610095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.610053 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" event={"ID":"f3d303af-cdff-4416-bdad-5f493d7c1343","Type":"ContainerStarted","Data":"1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc"} Oct 30 13:02:22.610244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.610130 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:22.611386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.611357 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" event={"ID":"fb37b59a-0c72-4523-886e-bd89e1412c14","Type":"ContainerStarted","Data":"8158a2233333e8f00ddaaba2d538eb343e63ef48fd4b341929dc799569cbbb42"} Oct 30 13:02:22.678909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:22.678797 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" podStartSLOduration=2.6787789159999997 podStartE2EDuration="2.678778916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:22.675889994 +0000 UTC m=+1760.802690893" watchObservedRunningTime="2026-10-30 13:02:22.678778916 +0000 UTC m=+1760.805579813" Oct 30 13:02:23.620011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:23.619973 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" event={"ID":"fb37b59a-0c72-4523-886e-bd89e1412c14","Type":"ContainerStarted","Data":"1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef"} Oct 30 13:02:23.620519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:23.620067 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:23.631368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:23.631336 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:23.673032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:23.672985 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" podStartSLOduration=2.315054063 podStartE2EDuration="2.672971331s" totalImagesPullingTime="357.917268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:23.670548779 +0000 UTC m=+1761.797349677" watchObservedRunningTime="2026-10-30 13:02:23.672971331 +0000 UTC m=+1761.799772229" Oct 30 13:02:24.168901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.167923 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:02:24.365853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.365767 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-credential-socket\") pod \"57f8ce64-4c88-4269-ab23-c96097bd9669\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " Oct 30 13:02:24.365853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.365805 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-token\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-token\") pod \"57f8ce64-4c88-4269-ab23-c96097bd9669\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " Oct 30 13:02:24.365853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.365837 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f8ce64-4c88-4269-ab23-c96097bd9669-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f8ce64-4c88-4269-ab23-c96097bd9669-istiod-ca-cert\") pod \"57f8ce64-4c88-4269-ab23-c96097bd9669\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " Oct 30 13:02:24.365853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.365854 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-kube-api-access-fhz7x\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-kube-api-access-fhz7x\") pod \"57f8ce64-4c88-4269-ab23-c96097bd9669\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " Oct 30 13:02:24.366183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.365947 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-podinfo\") pod \"57f8ce64-4c88-4269-ab23-c96097bd9669\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " Oct 30 13:02:24.366183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.366010 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-envoy\") pod \"57f8ce64-4c88-4269-ab23-c96097bd9669\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " Oct 30 13:02:24.366183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.366076 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-socket\") pod \"57f8ce64-4c88-4269-ab23-c96097bd9669\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " Oct 30 13:02:24.366183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.366149 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-credential-socket" pod "57f8ce64-4c88-4269-ab23-c96097bd9669" (UID: "57f8ce64-4c88-4269-ab23-c96097bd9669"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:24.366418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.366205 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-certs\") pod \"57f8ce64-4c88-4269-ab23-c96097bd9669\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " Oct 30 13:02:24.366418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.366255 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f8ce64-4c88-4269-ab23-c96097bd9669-istiod-ca-cert" pod "57f8ce64-4c88-4269-ab23-c96097bd9669" (UID: "57f8ce64-4c88-4269-ab23-c96097bd9669"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:24.366418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.366333 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-data\") pod \"57f8ce64-4c88-4269-ab23-c96097bd9669\" (UID: \"57f8ce64-4c88-4269-ab23-c96097bd9669\") " Oct 30 13:02:24.366555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.366523 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-socket" pod "57f8ce64-4c88-4269-ab23-c96097bd9669" (UID: "57f8ce64-4c88-4269-ab23-c96097bd9669"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:24.366869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.366835 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-data" pod "57f8ce64-4c88-4269-ab23-c96097bd9669" (UID: "57f8ce64-4c88-4269-ab23-c96097bd9669"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:24.367121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.367102 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:24.367181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.367129 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f8ce64-4c88-4269-ab23-c96097bd9669-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:24.367181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.367144 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:24.367181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.367158 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:24.367341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.367219 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-certs" pod "57f8ce64-4c88-4269-ab23-c96097bd9669" (UID: "57f8ce64-4c88-4269-ab23-c96097bd9669"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:24.368341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.368315 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-kube-api-access-fhz7x" pod "57f8ce64-4c88-4269-ab23-c96097bd9669" (UID: "57f8ce64-4c88-4269-ab23-c96097bd9669"). InnerVolumeSpecName "kube-api-access-fhz7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:24.368674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.368649 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-token" pod "57f8ce64-4c88-4269-ab23-c96097bd9669" (UID: "57f8ce64-4c88-4269-ab23-c96097bd9669"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:24.368756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.368676 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-podinfo" pod "57f8ce64-4c88-4269-ab23-c96097bd9669" (UID: "57f8ce64-4c88-4269-ab23-c96097bd9669"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:24.368756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.368729 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-envoy" pod "57f8ce64-4c88-4269-ab23-c96097bd9669" (UID: "57f8ce64-4c88-4269-ab23-c96097bd9669"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:24.471293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.470627 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:24.471293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.470667 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhz7x\" (UniqueName: \"kubernetes.io/projected/57f8ce64-4c88-4269-ab23-c96097bd9669-kube-api-access-fhz7x\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:24.471293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.470682 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:24.471293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.470695 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:24.471293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.470710 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f8ce64-4c88-4269-ab23-c96097bd9669-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:02:24.664286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.663609 5515 generic.go:356] "Generic (PLEG): container finished" podID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerID="0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a" exitCode=0 Oct 30 13:02:24.664769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.664745 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" Oct 30 13:02:24.683294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.682692 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" event={"ID":"57f8ce64-4c88-4269-ab23-c96097bd9669","Type":"ContainerDied","Data":"0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a"} Oct 30 13:02:24.683294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.682766 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4" event={"ID":"57f8ce64-4c88-4269-ab23-c96097bd9669","Type":"ContainerDied","Data":"417f543db47bffc0d6dd9b2aa7256c2525a2314733ce20b5ef891c68a0d0bb00"} Oct 30 13:02:24.683294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.682798 5515 scope.go:122] "RemoveContainer" containerID="0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a" Oct 30 13:02:24.762552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.748690 5515 scope.go:122] "RemoveContainer" containerID="489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1" Oct 30 13:02:24.789295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.788598 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4"] Oct 30 13:02:24.819292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.818876 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f399fe-openshift-default-645b7fcddc-k8zw4"] Oct 30 13:02:24.882962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.872431 5515 scope.go:122] "RemoveContainer" containerID="0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a" Oct 30 13:02:24.882962 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:02:24.872871 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a\": container with ID starting with 0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a not found: ID does not exist" containerID="0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a" Oct 30 13:02:24.882962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.872906 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a"} err="failed to get container status \"0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a\": rpc error: code = NotFound desc = could not find container \"0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a\": container with ID starting with 0a51302dbbaad6a8a7e02ae9ca64661b813a35de29c1b57b6ec0d1b857a8927a not found: ID does not exist" Oct 30 13:02:24.882962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.872931 5515 scope.go:122] "RemoveContainer" containerID="489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1" Oct 30 13:02:24.882962 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:02:24.873257 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1\": container with ID starting with 489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1 not found: ID does not exist" containerID="489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1" Oct 30 13:02:24.882962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:24.873300 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1"} err="failed to get container status \"489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1\": rpc error: code = NotFound desc = could not find container \"489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1\": container with ID starting with 489e8fae4b7fdf9dbb09a2c7a44763665ef0e45f53dba977a062bbd4eec5dab1 not found: ID does not exist" Oct 30 13:02:25.726306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:25.724535 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:26.036344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:26.029890 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" path="/var/lib/kubelet/pods/57f8ce64-4c88-4269-ab23-c96097bd9669/volumes" Oct 30 13:02:26.140300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:26.138360 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:02:27.689295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:27.685847 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:29.090117 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:02:29.090035 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:02:29.318840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:29.318805 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:29.669578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:29.669550 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:02:30.813170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:30.811990 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:32.364125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:32.362842 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:33.623623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:33.623600 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:02:33.814956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:33.814926 5515 generic.go:356] "Generic (PLEG): container finished" podID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerID="da085127e1616356c1e3f0a0c066b661d7f13ecc8b23383464f7439601af1938" exitCode=0 Oct 30 13:02:33.815118 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:33.815005 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" event={"ID":"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251","Type":"ContainerDied","Data":"da085127e1616356c1e3f0a0c066b661d7f13ecc8b23383464f7439601af1938"} Oct 30 13:02:33.815434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:33.815419 5515 scope.go:122] "RemoveContainer" containerID="da085127e1616356c1e3f0a0c066b661d7f13ecc8b23383464f7439601af1938" Oct 30 13:02:33.816874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:33.816841 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:33.816995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:33.816924 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:33.816995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:33.816971 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:33.919969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:33.919933 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:34.541290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:34.538447 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:34.829019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:34.828928 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" event={"ID":"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251","Type":"ContainerStarted","Data":"40a9296e7e21310e7ad78efeeb6907ce8316d3c364701db53b17c8ebc57c550f"} Oct 30 13:02:34.946506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:34.946473 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:02:34.946673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:34.946590 5515 patch_prober.go:29] interesting pod/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:02:34.946673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:34.946628 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:02:35.219162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.219124 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:35.455192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.455159 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:35.588220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.588115 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh"] Oct 30 13:02:35.589170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.589142 5515 cpu_manager.go:414] "containerMap: removing container" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" Oct 30 13:02:35.589170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.589170 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" Oct 30 13:02:35.589398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.589184 5515 cpu_manager.go:414] "containerMap: removing container" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" Oct 30 13:02:35.589398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.589192 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" Oct 30 13:02:35.589398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.589366 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" Oct 30 13:02:35.589398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.589381 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f8ce64-4c88-4269-ab23-c96097bd9669" containerName="istio-proxy" Oct 30 13:02:35.617289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.605524 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.634298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.631723 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh"] Oct 30 13:02:35.634298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.631919 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a2768f-openshift-default-dockercfg-hvk7c\"" Oct 30 13:02:35.639674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.639640 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e78981-662f-45c9-b066-e6cee6e117e8-istio-podinfo\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.639809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.639697 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-certs\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.639920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.639893 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e78981-662f-45c9-b066-e6cee6e117e8-istiod-ca-cert\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.639990 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.639941 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-credential-socket\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.640052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.640035 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-istio-token\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.640160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.640134 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkd9x\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-kube-api-access-dkd9x\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.640314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.640250 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-data\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.640389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.640370 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-socket\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.640450 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.640416 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-envoy\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.741572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.741536 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkd9x\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-kube-api-access-dkd9x\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.741760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.741650 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-data\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.741760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.741719 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-socket\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.741760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.741758 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-envoy\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.741938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.741810 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e78981-662f-45c9-b066-e6cee6e117e8-istio-podinfo\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.741938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.741871 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-certs\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.742038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.741976 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e78981-662f-45c9-b066-e6cee6e117e8-istiod-ca-cert\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.742038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.742009 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-credential-socket\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.742145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.742041 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-istio-token\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.742201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.742170 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-data\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.742201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.742190 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-socket\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.742456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.742436 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-certs\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.742620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.742602 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-credential-socket\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.742769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.742750 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e78981-662f-45c9-b066-e6cee6e117e8-istiod-ca-cert\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.744885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.744862 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e78981-662f-45c9-b066-e6cee6e117e8-istio-podinfo\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.745553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.745532 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-envoy\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.770907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.770883 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-istio-token\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.771405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.771388 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkd9x\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-kube-api-access-dkd9x\") pod \"e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:35.772868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.772830 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:35.952296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.948942 5515 patch_prober.go:29] interesting pod/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:02:35.952296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.949005 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:02:35.971329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:35.955507 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:36.080635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:36.079582 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:36.529619 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:02:36.529592 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e78981_662f_45c9_b066_e6cee6e117e8.slice/crio-36ea246f070e505a7d173114450effbd2a16403b18bf1a15ec13dead7fc4006a WatchSource:0}: Error finding container 36ea246f070e505a7d173114450effbd2a16403b18bf1a15ec13dead7fc4006a: Status 404 returned error can't find the container with id 36ea246f070e505a7d173114450effbd2a16403b18bf1a15ec13dead7fc4006a Oct 30 13:02:36.530368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:36.530345 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh"] Oct 30 13:02:36.532195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:36.532165 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:36.532279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:36.532229 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:36.532279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:36.532256 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:36.950412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:36.950378 5515 patch_prober.go:29] interesting pod/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:02:36.950684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:36.950655 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:02:36.966990 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:36.965831 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" event={"ID":"03e78981-662f-45c9-b066-e6cee6e117e8","Type":"ContainerStarted","Data":"cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838"} Oct 30 13:02:36.966990 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:36.965901 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" event={"ID":"03e78981-662f-45c9-b066-e6cee6e117e8","Type":"ContainerStarted","Data":"36ea246f070e505a7d173114450effbd2a16403b18bf1a15ec13dead7fc4006a"} Oct 30 13:02:37.179296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:37.172753 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:37.947385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:37.947347 5515 patch_prober.go:29] interesting pod/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:02:37.947568 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:37.947411 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:02:37.956069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:37.956039 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:37.962385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:37.962363 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:37.972307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:37.972288 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:37.973051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:37.973032 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:02:38.037653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:38.037603 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" podStartSLOduration=3.037590501 podStartE2EDuration="3.037590501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:37.039319936 +0000 UTC m=+1775.166120835" watchObservedRunningTime="2026-10-30 13:02:38.037590501 +0000 UTC m=+1776.164391399" Oct 30 13:02:38.952293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:38.949749 5515 patch_prober.go:29] interesting pod/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:02:38.952293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:38.949852 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:02:38.952293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:38.950306 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:39.586993 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:39.586962 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:39.820994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:39.820964 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:39.947154 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:39.947123 5515 patch_prober.go:29] interesting pod/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:02:39.947355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:39.947168 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:02:40.601997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:40.597518 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:40.951318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:40.951287 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:02:40.951699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:40.951654 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:02:40.952384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:40.952365 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:02:41.305373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.305264 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:41.412846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.412814 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9"] Oct 30 13:02:41.416084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.416068 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:41.434349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.433392 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a2768f-dockercfg-lrszl\"" Oct 30 13:02:41.434349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.433473 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a2768f\"" Oct 30 13:02:41.438075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.438049 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9"] Oct 30 13:02:41.526786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.526744 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72h8\" (UniqueName: \"kubernetes.io/projected/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-kube-api-access-q72h8\") pod \"payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9\" (UID: \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:41.537297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.529191 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9\" (UID: \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:41.777638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.767401 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:41.879233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.879201 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72h8\" (UniqueName: \"kubernetes.io/projected/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-kube-api-access-q72h8\") pod \"payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9\" (UID: \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:41.879428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.879403 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9\" (UID: \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:41.880117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.880094 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9\" (UID: \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:41.902678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:41.902647 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72h8\" (UniqueName: \"kubernetes.io/projected/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-kube-api-access-q72h8\") pod \"payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9\" (UID: \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:42.029582 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:42.029508 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:42.281987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:42.281826 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:42.350689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:42.350657 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9"] Oct 30 13:02:42.351378 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:02:42.351355 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff9dcd0_8e60_4bd3_b0a1_abc4a0146e60.slice/crio-7f70c4747c371aa703f2ebec0b156d04d8feb2cb9127221977bea75d9c8a5e2d WatchSource:0}: Error finding container 7f70c4747c371aa703f2ebec0b156d04d8feb2cb9127221977bea75d9c8a5e2d: Status 404 returned error can't find the container with id 7f70c4747c371aa703f2ebec0b156d04d8feb2cb9127221977bea75d9c8a5e2d Oct 30 13:02:43.056806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:43.056766 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" event={"ID":"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60","Type":"ContainerStarted","Data":"0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c"} Oct 30 13:02:43.057304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:43.056832 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" event={"ID":"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60","Type":"ContainerStarted","Data":"7f70c4747c371aa703f2ebec0b156d04d8feb2cb9127221977bea75d9c8a5e2d"} Oct 30 13:02:43.057304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:43.056858 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:43.101543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:43.101497 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" podStartSLOduration=2.101484487 podStartE2EDuration="2.101484487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:43.098031831 +0000 UTC m=+1781.224832729" watchObservedRunningTime="2026-10-30 13:02:43.101484487 +0000 UTC m=+1781.228285384" Oct 30 13:02:43.680442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:43.667997 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:02:43.934331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:43.934236 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:45.645359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:45.645171 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:45.963313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:45.963193 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:46.269482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.269404 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b"] Oct 30 13:02:46.274578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.274542 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.292133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.292110 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9368ce-openshift-default-dockercfg-vxchg\"" Oct 30 13:02:46.310084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.310053 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b"] Oct 30 13:02:46.408304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.401746 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-token\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.408619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.408587 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-socket\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.425449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.425410 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-credential-socket\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.425695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.425676 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-data\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.425912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.425894 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqlf5\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-kube-api-access-rqlf5\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.426137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.426118 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-podinfo\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.426303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.426287 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-certs\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.426434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.426420 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55811f26-0cc4-42d3-b727-8f1c8f757450-istiod-ca-cert\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.426565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.426552 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-envoy\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.461336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.442795 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.544577 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-credential-socket\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.544649 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-data\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.544730 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqlf5\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-kube-api-access-rqlf5\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.544770 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-podinfo\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.544807 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-certs\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.544831 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55811f26-0cc4-42d3-b727-8f1c8f757450-istiod-ca-cert\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.544878 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-envoy\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.545053 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-token\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.545092 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-socket\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.545646 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-socket\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.545941 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-credential-socket\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.550367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.546184 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-data\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.577314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.555776 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55811f26-0cc4-42d3-b727-8f1c8f757450-istiod-ca-cert\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.577314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.567152 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-certs\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.610852 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.610732 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-envoy\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.613481 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.613449 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-podinfo\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.615059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.615001 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-token\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.637877 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.637845 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqlf5\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-kube-api-access-rqlf5\") pod \"e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.694785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.694753 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:46.891727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.891696 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:46.924405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:46.924377 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:47.198303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:47.197380 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:47.255323 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:02:47.255292 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55811f26_0cc4_42d3_b727_8f1c8f757450.slice/crio-c4beb4c83d81073022f2ba0b0eb434a7623541108ade2fe635d4126cf84ec7ec WatchSource:0}: Error finding container c4beb4c83d81073022f2ba0b0eb434a7623541108ade2fe635d4126cf84ec7ec: Status 404 returned error can't find the container with id c4beb4c83d81073022f2ba0b0eb434a7623541108ade2fe635d4126cf84ec7ec Oct 30 13:02:47.256940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:47.256912 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b"] Oct 30 13:02:47.259043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:47.258918 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:47.259043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:47.259005 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:47.259205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:47.259053 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:02:47.526610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:47.526523 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:48.100544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:48.100510 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" event={"ID":"55811f26-0cc4-42d3-b727-8f1c8f757450","Type":"ContainerStarted","Data":"626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925"} Oct 30 13:02:48.100910 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:48.100560 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" event={"ID":"55811f26-0cc4-42d3-b727-8f1c8f757450","Type":"ContainerStarted","Data":"c4beb4c83d81073022f2ba0b0eb434a7623541108ade2fe635d4126cf84ec7ec"} Oct 30 13:02:48.158377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:48.158324 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" podStartSLOduration=2.158310991 podStartE2EDuration="2.158310991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:48.157400893 +0000 UTC m=+1786.284201792" watchObservedRunningTime="2026-10-30 13:02:48.158310991 +0000 UTC m=+1786.285111889" Oct 30 13:02:48.892821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:48.892025 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:48.904584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:48.904537 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:48.909579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:48.909550 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:49.107665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:49.107633 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:49.108701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:49.108678 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:02:50.431310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:50.423732 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:50.690151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:50.690071 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:50.933817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:50.933785 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:51.167763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:51.167729 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:51.671328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:51.671297 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:52.381359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:52.381324 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:52.665291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:52.659919 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.268336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:53.266991 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.505685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:53.505651 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:54.068101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:54.068073 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:02:54.866201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:54.866165 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:56.323060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:56.322969 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:56.564760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:56.564729 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:57.619654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:57.619621 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r"] Oct 30 13:02:57.780160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:57.780131 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:58.005129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:58.005088 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:58.510337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:58.510261 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:58.739105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:58.739075 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:58.994860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:58.994797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:59.066712 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:02:59.066677 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:02:59.219944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:59.219915 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:59.450852 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:59.450821 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:59.806074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:02:59.805993 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.309292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.284217 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t"] Oct 30 13:03:00.322845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.322815 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t"] Oct 30 13:03:00.323211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.323134 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.353643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.350105 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6f0f50-openshift-default-dockercfg-k7fg8\"" Oct 30 13:03:00.420700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.410020 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g4kd\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-kube-api-access-8g4kd\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.420700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.419509 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-token\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.420700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.419616 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-envoy\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.420700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.419701 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-podinfo\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.420700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.419888 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-socket\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.420700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.419971 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-certs\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.420700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.420007 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istiod-ca-cert\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.420700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.420066 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-data\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.420700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.420112 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-credential-socket\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.600122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.597382 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-envoy\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.600122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.597487 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-podinfo\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.600122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.597617 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-socket\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.600122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.597680 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-certs\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.600122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.597714 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istiod-ca-cert\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.600122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.597754 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-data\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.600122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.597788 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-credential-socket\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.600122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.597822 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g4kd\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-kube-api-access-8g4kd\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.600122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.597847 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-token\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.606538 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.601620 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.606538 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.602373 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istiod-ca-cert\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.621241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.607839 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-socket\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.622189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.622158 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-credential-socket\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.622647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.622627 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-data\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.623344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.623324 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-certs\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.700107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.692170 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-token\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.709254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.708878 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-podinfo\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.709254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.709191 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-envoy\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.717878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.717842 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g4kd\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-kube-api-access-8g4kd\") pod \"e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:00.818979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.818930 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.987326 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:00.987293 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:01.233053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:01.219731 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:01.317652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:01.317620 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t"] Oct 30 13:03:01.317765 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:03:01.317635 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f3746b_ffc4_4523_94a4_4c68b3eaa397.slice/crio-3250e73304e9e9ebed83c568356352811ca4e65040f10e6cdd256e7fc72f1295 WatchSource:0}: Error finding container 3250e73304e9e9ebed83c568356352811ca4e65040f10e6cdd256e7fc72f1295: Status 404 returned error can't find the container with id 3250e73304e9e9ebed83c568356352811ca4e65040f10e6cdd256e7fc72f1295 Oct 30 13:03:01.320470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:01.320430 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:01.320567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:01.320533 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:01.320606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:01.320574 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:02.284732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:02.284699 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" event={"ID":"f9f3746b-ffc4-4523-94a4-4c68b3eaa397","Type":"ContainerStarted","Data":"cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc"} Oct 30 13:03:02.285196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:02.284750 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" event={"ID":"f9f3746b-ffc4-4523-94a4-4c68b3eaa397","Type":"ContainerStarted","Data":"3250e73304e9e9ebed83c568356352811ca4e65040f10e6cdd256e7fc72f1295"} Oct 30 13:03:02.349908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:02.349854 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" podStartSLOduration=2.349836544 podStartE2EDuration="2.349836544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:02.348916075 +0000 UTC m=+1800.475716973" watchObservedRunningTime="2026-10-30 13:03:02.349836544 +0000 UTC m=+1800.476637444" Oct 30 13:03:02.620680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:02.620595 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" podUID="f3d303af-cdff-4416-bdad-5f493d7c1343" containerName="payload-processing" containerID="cri-o://1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc" gracePeriod=25 Oct 30 13:03:02.998411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:02.998376 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:03.006430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.006403 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:03.008260 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.008244 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:03:03.029831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.029806 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.051921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.051891 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d303af-cdff-4416-bdad-5f493d7c1343-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d303af-cdff-4416-bdad-5f493d7c1343-plugins-config-volume\") pod \"f3d303af-cdff-4416-bdad-5f493d7c1343\" (UID: \"f3d303af-cdff-4416-bdad-5f493d7c1343\") " Oct 30 13:03:03.052082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.052046 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d303af-cdff-4416-bdad-5f493d7c1343-kube-api-access-w9q6c\" (UniqueName: \"kubernetes.io/projected/f3d303af-cdff-4416-bdad-5f493d7c1343-kube-api-access-w9q6c\") pod \"f3d303af-cdff-4416-bdad-5f493d7c1343\" (UID: \"f3d303af-cdff-4416-bdad-5f493d7c1343\") " Oct 30 13:03:03.052832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.052306 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d303af-cdff-4416-bdad-5f493d7c1343-plugins-config-volume" pod "f3d303af-cdff-4416-bdad-5f493d7c1343" (UID: "f3d303af-cdff-4416-bdad-5f493d7c1343"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:03.052832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.052763 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d303af-cdff-4416-bdad-5f493d7c1343-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:03.054110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.054089 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d303af-cdff-4416-bdad-5f493d7c1343-kube-api-access-w9q6c" pod "f3d303af-cdff-4416-bdad-5f493d7c1343" (UID: "f3d303af-cdff-4416-bdad-5f493d7c1343"). InnerVolumeSpecName "kube-api-access-w9q6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:03.153567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.153532 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9q6c\" (UniqueName: \"kubernetes.io/projected/f3d303af-cdff-4416-bdad-5f493d7c1343-kube-api-access-w9q6c\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:03.293447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.293367 5515 generic.go:356] "Generic (PLEG): container finished" podID="f3d303af-cdff-4416-bdad-5f493d7c1343" containerID="1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc" exitCode=0 Oct 30 13:03:03.293447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.293412 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" Oct 30 13:03:03.293905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.293452 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" event={"ID":"f3d303af-cdff-4416-bdad-5f493d7c1343","Type":"ContainerDied","Data":"1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc"} Oct 30 13:03:03.293905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.293502 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r" event={"ID":"f3d303af-cdff-4416-bdad-5f493d7c1343","Type":"ContainerDied","Data":"a3cbfcec90ab03f4320c37c9c3325ea720feb70e93f92f22a2bb46fffa0985df"} Oct 30 13:03:03.293905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.293533 5515 scope.go:122] "RemoveContainer" containerID="1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc" Oct 30 13:03:03.294037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.293908 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:03.294990 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.294972 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:03.303725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.303709 5515 scope.go:122] "RemoveContainer" containerID="1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc" Oct 30 13:03:03.303982 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:03:03.303960 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc\": container with ID starting with 1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc not found: ID does not exist" containerID="1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc" Oct 30 13:03:03.304046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.303991 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc"} err="failed to get container status \"1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc\": rpc error: code = NotFound desc = could not find container \"1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc\": container with ID starting with 1e41dac5e167f4629a8ae6bbe53b48ab6e58fdf0c23b11fae8e58cf14daa0adc not found: ID does not exist" Oct 30 13:03:03.348726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.348700 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r"] Oct 30 13:03:03.352134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:03.352111 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ea117f29-67d879c59b-wrh5r"] Oct 30 13:03:04.003166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.003134 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d303af-cdff-4416-bdad-5f493d7c1343" path="/var/lib/kubelet/pods/f3d303af-cdff-4416-bdad-5f493d7c1343/volumes" Oct 30 13:03:04.390658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.390625 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd"] Oct 30 13:03:04.391123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.390857 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" podUID="fb37b59a-0c72-4523-886e-bd89e1412c14" containerName="maas-api" containerID="cri-o://1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef" gracePeriod=30 Oct 30 13:03:04.598975 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.598936 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:04.874293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.873457 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:03:04.898242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.898208 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-maas-api-tls\") pod \"fb37b59a-0c72-4523-886e-bd89e1412c14\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " Oct 30 13:03:04.898429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.898312 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-metrics-certs\") pod \"fb37b59a-0c72-4523-886e-bd89e1412c14\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " Oct 30 13:03:04.898429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.898418 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb37b59a-0c72-4523-886e-bd89e1412c14-kube-api-access-mckbt\" (UniqueName: \"kubernetes.io/projected/fb37b59a-0c72-4523-886e-bd89e1412c14-kube-api-access-mckbt\") pod \"fb37b59a-0c72-4523-886e-bd89e1412c14\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " Oct 30 13:03:04.898504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.898448 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb37b59a-0c72-4523-886e-bd89e1412c14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb37b59a-0c72-4523-886e-bd89e1412c14-openshift-service-ca\") pod \"fb37b59a-0c72-4523-886e-bd89e1412c14\" (UID: \"fb37b59a-0c72-4523-886e-bd89e1412c14\") " Oct 30 13:03:04.899357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.899328 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb37b59a-0c72-4523-886e-bd89e1412c14-openshift-service-ca" pod "fb37b59a-0c72-4523-886e-bd89e1412c14" (UID: "fb37b59a-0c72-4523-886e-bd89e1412c14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:04.901594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.901556 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb37b59a-0c72-4523-886e-bd89e1412c14-kube-api-access-mckbt" pod "fb37b59a-0c72-4523-886e-bd89e1412c14" (UID: "fb37b59a-0c72-4523-886e-bd89e1412c14"). InnerVolumeSpecName "kube-api-access-mckbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:04.901732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.901717 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-maas-api-tls" pod "fb37b59a-0c72-4523-886e-bd89e1412c14" (UID: "fb37b59a-0c72-4523-886e-bd89e1412c14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:04.901804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.901744 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-metrics-certs" pod "fb37b59a-0c72-4523-886e-bd89e1412c14" (UID: "fb37b59a-0c72-4523-886e-bd89e1412c14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:04.936753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.936722 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:04.997426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.997346 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz"] Oct 30 13:03:04.998045 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.998030 5515 cpu_manager.go:414] "containerMap: removing container" podUID="fb37b59a-0c72-4523-886e-bd89e1412c14" containerName="maas-api" Oct 30 13:03:04.998082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.998047 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb37b59a-0c72-4523-886e-bd89e1412c14" containerName="maas-api" Oct 30 13:03:04.998082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.998067 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f3d303af-cdff-4416-bdad-5f493d7c1343" containerName="payload-processing" Oct 30 13:03:04.998082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.998072 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d303af-cdff-4416-bdad-5f493d7c1343" containerName="payload-processing" Oct 30 13:03:04.998201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.998156 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb37b59a-0c72-4523-886e-bd89e1412c14" containerName="maas-api" Oct 30 13:03:04.998201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:04.998167 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d303af-cdff-4416-bdad-5f493d7c1343" containerName="payload-processing" Oct 30 13:03:05.000799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.000776 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mckbt\" (UniqueName: \"kubernetes.io/projected/fb37b59a-0c72-4523-886e-bd89e1412c14-kube-api-access-mckbt\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:05.000862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.000804 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb37b59a-0c72-4523-886e-bd89e1412c14-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:05.000862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.000818 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:05.000862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.000830 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb37b59a-0c72-4523-886e-bd89e1412c14-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:05.001956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.001934 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.023129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.022447 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9368ce-serving-cert\"" Oct 30 13:03:05.028421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.028213 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz"] Oct 30 13:03:05.101698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.101661 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.101921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.101899 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-metrics-certs\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.102031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.101987 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61452246-0bb8-4c65-bdbd-b2c3f53977cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.102090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.102033 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdw85\" (UniqueName: \"kubernetes.io/projected/61452246-0bb8-4c65-bdbd-b2c3f53977cb-kube-api-access-xdw85\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.191811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.191717 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:05.231292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.208551 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-metrics-certs\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.231292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.208656 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61452246-0bb8-4c65-bdbd-b2c3f53977cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.231292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.208704 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdw85\" (UniqueName: \"kubernetes.io/projected/61452246-0bb8-4c65-bdbd-b2c3f53977cb-kube-api-access-xdw85\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.231292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.208802 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.231292 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:03:05.209093 5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-serving-cert: secret "maas-api-e2e-ipp-a-9368ce-serving-cert" not found Oct 30 13:03:05.231292 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:03:05.209180 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls podName:61452246-0bb8-4c65-bdbd-b2c3f53977cb nodeName:}" failed. No retries permitted until 2026-10-30 13:03:05.709154856 +0000 UTC m=+1803.835955755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls") pod "maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" (UID: "61452246-0bb8-4c65-bdbd-b2c3f53977cb") : secret "maas-api-e2e-ipp-a-9368ce-serving-cert" not found Oct 30 13:03:05.231292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.221712 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61452246-0bb8-4c65-bdbd-b2c3f53977cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.242255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.242183 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-metrics-certs\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.278341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.278048 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdw85\" (UniqueName: \"kubernetes.io/projected/61452246-0bb8-4c65-bdbd-b2c3f53977cb-kube-api-access-xdw85\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.367926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.360087 5515 generic.go:356] "Generic (PLEG): container finished" podID="fb37b59a-0c72-4523-886e-bd89e1412c14" containerID="1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef" exitCode=0 Oct 30 13:03:05.367926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.361526 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" Oct 30 13:03:05.397649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.388958 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" event={"ID":"fb37b59a-0c72-4523-886e-bd89e1412c14","Type":"ContainerDied","Data":"1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef"} Oct 30 13:03:05.397649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.389035 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" event={"ID":"fb37b59a-0c72-4523-886e-bd89e1412c14","Type":"ContainerDied","Data":"8158a2233333e8f00ddaaba2d538eb343e63ef48fd4b341929dc799569cbbb42"} Oct 30 13:03:05.397649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.389067 5515 scope.go:122] "RemoveContainer" containerID="1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef" Oct 30 13:03:05.471521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.460226 5515 scope.go:122] "RemoveContainer" containerID="1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef" Oct 30 13:03:05.471521 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:03:05.461135 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef\": container with ID starting with 1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef not found: ID does not exist" containerID="1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef" Oct 30 13:03:05.471521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.461176 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef"} err="failed to get container status \"1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef\": rpc error: code = NotFound desc = could not find container \"1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef\": container with ID starting with 1aa83f9058303b761aabbd20c5c0ae2a51efec9dfa0df89eeb649eb2c75be0ef not found: ID does not exist" Oct 30 13:03:05.506388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.506321 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd"] Oct 30 13:03:05.517702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.517664 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd"] Oct 30 13:03:05.523292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.523179 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:05.726505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.726470 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.728946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.728913 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:05.789106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.789074 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:05.917590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:05.917561 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:06.005778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:06.005751 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb37b59a-0c72-4523-886e-bd89e1412c14" path="/var/lib/kubelet/pods/fb37b59a-0c72-4523-886e-bd89e1412c14/volumes" Oct 30 13:03:06.260321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:06.260219 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:06.276463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:06.276424 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz"] Oct 30 13:03:06.279558 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:03:06.279535 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61452246_0bb8_4c65_bdbd_b2c3f53977cb.slice/crio-f99c1c41a825cea72a9d9b50c50ca591a843545c64ff524252a0124ba5426617 WatchSource:0}: Error finding container f99c1c41a825cea72a9d9b50c50ca591a843545c64ff524252a0124ba5426617: Status 404 returned error can't find the container with id f99c1c41a825cea72a9d9b50c50ca591a843545c64ff524252a0124ba5426617 Oct 30 13:03:06.369200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:06.369167 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" event={"ID":"61452246-0bb8-4c65-bdbd-b2c3f53977cb","Type":"ContainerStarted","Data":"f99c1c41a825cea72a9d9b50c50ca591a843545c64ff524252a0124ba5426617"} Oct 30 13:03:06.652906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:06.652878 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.070339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:07.064957 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.377529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:07.377437 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" event={"ID":"61452246-0bb8-4c65-bdbd-b2c3f53977cb","Type":"ContainerStarted","Data":"a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05"} Oct 30 13:03:07.377665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:07.377588 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:07.454829 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:07.451337 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" podStartSLOduration=3.0148475 podStartE2EDuration="3.45131603s" totalImagesPullingTime="436.46853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:07.449807284 +0000 UTC m=+1805.576608184" watchObservedRunningTime="2026-10-30 13:03:07.45131603 +0000 UTC m=+1805.578116929" Oct 30 13:03:07.524805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:07.524603 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.694142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:07.694090 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-rc4zd" podUID="fb37b59a-0c72-4523-886e-bd89e1412c14" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.102:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:08.345566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:08.345533 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:09.303807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:09.303778 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:09.893696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:09.893668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:10.852192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:10.850169 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:11.089977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:11.089945 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf"] Oct 30 13:03:11.337917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:11.337886 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:12.533994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:12.533908 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:13.390142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:13.390112 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:03:14.109138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:14.109104 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:14.677670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:14.677637 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:15.175607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:15.175576 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:15.972657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:15.972629 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:17.605616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:17.605579 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:19.154967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:19.154925 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:20.401407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.401375 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j"] Oct 30 13:03:20.406976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.406959 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.424167 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.424141 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6f0f50-serving-cert\"" Oct 30 13:03:20.441416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.441386 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j"] Oct 30 13:03:20.500411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.500374 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428pf\" (UniqueName: \"kubernetes.io/projected/c15fd225-5060-4b06-95ce-682e90d25cdc-kube-api-access-428pf\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.500575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.500439 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-metrics-certs\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.500575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.500484 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15fd225-5060-4b06-95ce-682e90d25cdc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.500790 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.500766 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.602360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.602322 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.611826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.611789 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428pf\" (UniqueName: \"kubernetes.io/projected/c15fd225-5060-4b06-95ce-682e90d25cdc-kube-api-access-428pf\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.612187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.612163 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-metrics-certs\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.612487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.612465 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15fd225-5060-4b06-95ce-682e90d25cdc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.635340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.613682 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15fd225-5060-4b06-95ce-682e90d25cdc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.635601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.623187 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.635601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.633782 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-metrics-certs\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.635839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.635194 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:20.651140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.651104 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-428pf\" (UniqueName: \"kubernetes.io/projected/c15fd225-5060-4b06-95ce-682e90d25cdc-kube-api-access-428pf\") pod \"maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.721883 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.721758 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:20.882252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:20.882225 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.033468 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:03:21.033428 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15fd225_5060_4b06_95ce_682e90d25cdc.slice/crio-79addd98a19a3ca664854f747990b39a3e5aae1ba2f5ea1c07e9a8efc2e593d8 WatchSource:0}: Error finding container 79addd98a19a3ca664854f747990b39a3e5aae1ba2f5ea1c07e9a8efc2e593d8: Status 404 returned error can't find the container with id 79addd98a19a3ca664854f747990b39a3e5aae1ba2f5ea1c07e9a8efc2e593d8 Oct 30 13:03:21.033468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:21.033435 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j"] Oct 30 13:03:21.612298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:21.601637 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" event={"ID":"c15fd225-5060-4b06-95ce-682e90d25cdc","Type":"ContainerStarted","Data":"1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712"} Oct 30 13:03:21.612298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:21.601703 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" event={"ID":"c15fd225-5060-4b06-95ce-682e90d25cdc","Type":"ContainerStarted","Data":"79addd98a19a3ca664854f747990b39a3e5aae1ba2f5ea1c07e9a8efc2e593d8"} Oct 30 13:03:21.612298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:21.602836 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:21.648456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:21.647862 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" podStartSLOduration=1.312358405 podStartE2EDuration="1.647847189s" totalImagesPullingTime="335.488784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:21.64623034 +0000 UTC m=+1819.773031238" watchObservedRunningTime="2026-10-30 13:03:21.647847189 +0000 UTC m=+1819.774648086" Oct 30 13:03:22.299573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:22.299536 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:24.021290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:24.014176 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:25.608295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:25.607693 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.110784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.110747 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" podUID="a97c75bb-9fae-4978-a460-39b24d898ad5" containerName="main" containerID="cri-o://a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be" gracePeriod=45 Oct 30 13:03:26.277308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.277255 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" podUID="a97c75bb-9fae-4978-a460-39b24d898ad5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": dial tcp 10.134.0.96:8000: connect: connection refused" Oct 30 13:03:26.390432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.390404 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:03:26.431179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.431148 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97c75bb-9fae-4978-a460-39b24d898ad5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a97c75bb-9fae-4978-a460-39b24d898ad5-tls-certs\") pod \"a97c75bb-9fae-4978-a460-39b24d898ad5\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " Oct 30 13:03:26.431423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.431398 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-tmp-dir\") pod \"a97c75bb-9fae-4978-a460-39b24d898ad5\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " Oct 30 13:03:26.432399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.431481 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97c75bb-9fae-4978-a460-39b24d898ad5-kube-api-access-hzcz5\" (UniqueName: \"kubernetes.io/projected/a97c75bb-9fae-4978-a460-39b24d898ad5-kube-api-access-hzcz5\") pod \"a97c75bb-9fae-4978-a460-39b24d898ad5\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " Oct 30 13:03:26.432399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.431563 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-dshm\") pod \"a97c75bb-9fae-4978-a460-39b24d898ad5\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " Oct 30 13:03:26.432399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.431616 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-model-cache\") pod \"a97c75bb-9fae-4978-a460-39b24d898ad5\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " Oct 30 13:03:26.432399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.431850 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-model-cache" pod "a97c75bb-9fae-4978-a460-39b24d898ad5" (UID: "a97c75bb-9fae-4978-a460-39b24d898ad5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:26.432399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.432092 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-home\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-home\") pod \"a97c75bb-9fae-4978-a460-39b24d898ad5\" (UID: \"a97c75bb-9fae-4978-a460-39b24d898ad5\") " Oct 30 13:03:26.432399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.432095 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-home" pod "a97c75bb-9fae-4978-a460-39b24d898ad5" (UID: "a97c75bb-9fae-4978-a460-39b24d898ad5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:26.432877 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.432852 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:26.432992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.432880 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:26.433239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.433218 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-tmp-dir" pod "a97c75bb-9fae-4978-a460-39b24d898ad5" (UID: "a97c75bb-9fae-4978-a460-39b24d898ad5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:26.434571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.434552 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97c75bb-9fae-4978-a460-39b24d898ad5-tls-certs" pod "a97c75bb-9fae-4978-a460-39b24d898ad5" (UID: "a97c75bb-9fae-4978-a460-39b24d898ad5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:26.434824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.434807 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97c75bb-9fae-4978-a460-39b24d898ad5-kube-api-access-hzcz5" pod "a97c75bb-9fae-4978-a460-39b24d898ad5" (UID: "a97c75bb-9fae-4978-a460-39b24d898ad5"). InnerVolumeSpecName "kube-api-access-hzcz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:26.435363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.435334 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-dshm" pod "a97c75bb-9fae-4978-a460-39b24d898ad5" (UID: "a97c75bb-9fae-4978-a460-39b24d898ad5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:26.534509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.534482 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97c75bb-9fae-4978-a460-39b24d898ad5-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:26.534509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.534505 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:26.534509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.534516 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzcz5\" (UniqueName: \"kubernetes.io/projected/a97c75bb-9fae-4978-a460-39b24d898ad5-kube-api-access-hzcz5\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:26.534729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.534524 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97c75bb-9fae-4978-a460-39b24d898ad5-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:03:26.592458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.592429 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.723368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.717802 5515 generic.go:356] "Generic (PLEG): container finished" podID="a97c75bb-9fae-4978-a460-39b24d898ad5" containerID="a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be" exitCode=0 Oct 30 13:03:26.723368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.718057 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" event={"ID":"a97c75bb-9fae-4978-a460-39b24d898ad5","Type":"ContainerDied","Data":"a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be"} Oct 30 13:03:26.723368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.718092 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" event={"ID":"a97c75bb-9fae-4978-a460-39b24d898ad5","Type":"ContainerDied","Data":"ab8d540c22809ac59852589dc75ea6a15a04a0b75a6e3a3a73e6925efea65ce5"} Oct 30 13:03:26.723368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.718113 5515 scope.go:122] "RemoveContainer" containerID="a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be" Oct 30 13:03:26.724754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.718261 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf" Oct 30 13:03:26.765738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.765711 5515 scope.go:122] "RemoveContainer" containerID="a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be" Oct 30 13:03:26.766293 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:03:26.766143 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be\": container with ID starting with a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be not found: ID does not exist" containerID="a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be" Oct 30 13:03:26.766293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.766174 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be"} err="failed to get container status \"a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be\": rpc error: code = NotFound desc = could not find container \"a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be\": container with ID starting with a8d2085ee57324b24ac353c9d91350aedec7f7185a7768ac30080e467bdc25be not found: ID does not exist" Oct 30 13:03:26.799827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.799802 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf"] Oct 30 13:03:26.804531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.804511 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ea117f29/e2e-lifecycle-model-ea117f29-kserve-5d7b46c77c-znwsf"] Oct 30 13:03:26.830183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:26.830160 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.208966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:27.208853 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.570448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:27.570357 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.729335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:27.729296 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:03:27.790944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:27.790915 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:28.161360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:28.153764 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97c75bb-9fae-4978-a460-39b24d898ad5" path="/var/lib/kubelet/pods/a97c75bb-9fae-4978-a460-39b24d898ad5/volumes" Oct 30 13:03:28.641913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:28.641889 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:03:28.806954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:28.806032 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:29.072702 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:03:29.072615 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:03:30.339346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:30.337365 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:31.979292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:31.974861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:33.506795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:33.501906 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:34.986850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:34.986820 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:36.664754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:36.664717 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:37.887053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:37.887018 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:38.123297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:38.112771 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:39.016209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:39.016115 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:40.584570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:40.584524 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:40.826725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:40.826687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:42.327336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:42.323010 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:43.880215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:43.880180 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:45.377292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:45.365431 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:46.884904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:46.884841 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:48.052060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:48.051932 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:48.288414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:48.288250 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:48.661039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:48.660913 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:48.959564 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:48.959487 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:49.335504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:49.335418 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:03:49.335946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:49.335506 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:03:49.353134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:49.353096 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:49.601717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:49.601299 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": read tcp 10.134.0.2:44134->10.134.0.105:15021: read: connection reset by peer" start-of-body= Oct 30 13:03:49.601717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:49.601359 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": read tcp 10.134.0.2:44134->10.134.0.105:15021: read: connection reset by peer" Oct 30 13:03:50.043380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:50.043348 5515 generic.go:356] "Generic (PLEG): container finished" podID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerID="626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925" exitCode=0 Oct 30 13:03:50.043549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:50.043398 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" event={"ID":"55811f26-0cc4-42d3-b727-8f1c8f757450","Type":"ContainerDied","Data":"626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925"} Oct 30 13:03:50.043843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:50.043827 5515 scope.go:122] "RemoveContainer" containerID="626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925" Oct 30 13:03:50.045879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:50.045839 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:50.046018 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:50.045931 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:50.046018 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:50.045972 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:50.307647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:50.307549 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:50.819297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:50.817264 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:51.052985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:51.052948 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" event={"ID":"55811f26-0cc4-42d3-b727-8f1c8f757450","Type":"ContainerStarted","Data":"b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec"} Oct 30 13:03:51.644292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:51.641726 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:51.923344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:51.892232 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:03:51.923344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:51.892329 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:03:51.923344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:51.892384 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 13:03:52.015293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:52.013813 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:52.735301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:52.732733 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:52.921338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:52.920448 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:03:52.921338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:52.920544 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 13:03:53.252734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:53.252704 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:03:53.279349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:53.279321 5515 generic.go:356] "Generic (PLEG): container finished" podID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerID="cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc" exitCode=0 Oct 30 13:03:53.279516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:53.279371 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" event={"ID":"f9f3746b-ffc4-4523-94a4-4c68b3eaa397","Type":"ContainerDied","Data":"cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc"} Oct 30 13:03:53.279846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:53.279821 5515 scope.go:122] "RemoveContainer" containerID="cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc" Oct 30 13:03:53.281469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:53.281440 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:53.281549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:53.281508 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:53.281549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:53.281539 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:03:53.897293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:53.896649 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:03:53.897293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:53.896711 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 13:03:54.287929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:54.287838 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" event={"ID":"f9f3746b-ffc4-4523-94a4-4c68b3eaa397","Type":"ContainerStarted","Data":"c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30"} Oct 30 13:03:54.875325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:54.875284 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:54.893302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:54.893231 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:03:54.893615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:54.893585 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 13:03:55.041395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:55.041343 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:03:55.051694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:55.051661 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:03:55.051962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:55.051933 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 13:03:55.892214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:55.892184 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:03:55.892655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:55.892230 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 13:03:55.988671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:55.988637 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:03:55.989101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:55.988977 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 13:03:56.075519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:56.075484 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:56.267346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:56.267035 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd"] Oct 30 13:03:56.268098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:56.268042 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" containerID="cri-o://40a9296e7e21310e7ad78efeeb6907ce8316d3c364701db53b17c8ebc57c550f" gracePeriod=30 Oct 30 13:03:56.353537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:56.353502 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:56.610344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:56.609882 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:56.892834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:56.892801 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 13:03:56.893209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:56.892847 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 13:03:56.988426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:56.988390 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:03:56.988596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:56.988449 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 13:03:57.909296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:57.907231 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:03:57.909296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:57.908702 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:03:57.920292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:57.916484 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:03:57.988637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:57.988597 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:03:57.988817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:57.988665 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 13:03:58.125419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:58.125383 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:58.430519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:58.430478 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:58.936295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:58.932848 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:59.008468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:59.008419 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:03:59.008774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:59.008725 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 13:03:59.181941 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:59.181901 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:59.189704 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:03:59.189591 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:03:59.417121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:59.417071 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:59.907295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:59.898683 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:59.988150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:59.988116 5515 patch_prober.go:29] interesting pod/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 13:03:59.988572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:03:59.988167 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 13:04:00.191673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:00.191598 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:00.702158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:00.702120 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:00.927288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:00.925041 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:01.030655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.030578 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:04:01.032795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.032771 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:04:01.040487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.040447 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:04:01.488844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.485007 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:01.488844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.487251 5515 generic.go:356] "Generic (PLEG): container finished" podID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerID="40a9296e7e21310e7ad78efeeb6907ce8316d3c364701db53b17c8ebc57c550f" exitCode=0 Oct 30 13:04:01.488844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.487436 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" event={"ID":"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251","Type":"ContainerDied","Data":"40a9296e7e21310e7ad78efeeb6907ce8316d3c364701db53b17c8ebc57c550f"} Oct 30 13:04:01.488844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.487509 5515 scope.go:122] "RemoveContainer" containerID="da085127e1616356c1e3f0a0c066b661d7f13ecc8b23383464f7439601af1938" Oct 30 13:04:01.651843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.651816 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:04:01.762801 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.762762 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-podinfo\") pod \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " Oct 30 13:04:01.763314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.763256 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istiod-ca-cert\") pod \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " Oct 30 13:04:01.763551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.763514 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-certs\") pod \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " Oct 30 13:04:01.763860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.763729 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istiod-ca-cert" pod "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" (UID: "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:01.763860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.763840 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-certs" pod "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" (UID: "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:01.764010 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.763742 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-kube-api-access-lmtls\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-kube-api-access-lmtls\") pod \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " Oct 30 13:04:01.764010 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.763980 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-credential-socket\") pod \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " Oct 30 13:04:01.764130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.764060 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-token\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-token\") pod \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " Oct 30 13:04:01.764130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.764100 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-data\") pod \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " Oct 30 13:04:01.764242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.764140 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-socket\") pod \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " Oct 30 13:04:01.764314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.764237 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-envoy\") pod \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\" (UID: \"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251\") " Oct 30 13:04:01.764314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.764245 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-credential-socket" pod "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" (UID: "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:01.764497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.764473 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-data" pod "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" (UID: "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:01.764741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.764719 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-socket" pod "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" (UID: "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:01.765251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.765226 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:01.765398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.765383 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:01.765503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.765490 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:01.772673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.771515 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:01.772673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.767891 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-podinfo" pod "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" (UID: "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:01.772673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.768185 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-kube-api-access-lmtls" pod "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" (UID: "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251"). InnerVolumeSpecName "kube-api-access-lmtls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:01.772673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.768631 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-token" pod "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" (UID: "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:01.772673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.771600 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:01.772673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.767789 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-envoy" pod "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" (UID: "5e0cbb8a-8a4c-440a-923b-3aa8a52ac251"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:01.875528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.874970 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:01.875528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.875009 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmtls\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-kube-api-access-lmtls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:01.875528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.875025 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:01.875528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:01.875045 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:02.506166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:02.506137 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" Oct 30 13:04:02.506735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:02.506691 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd" event={"ID":"5e0cbb8a-8a4c-440a-923b-3aa8a52ac251","Type":"ContainerDied","Data":"288c81e7087d29b537a58078d9a4aca7b0aaa16823f59d02a7a3a8227c32506b"} Oct 30 13:04:02.506926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:02.506898 5515 scope.go:122] "RemoveContainer" containerID="40a9296e7e21310e7ad78efeeb6907ce8316d3c364701db53b17c8ebc57c550f" Oct 30 13:04:02.570753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:02.570722 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd"] Oct 30 13:04:02.582449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:02.582416 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f60f16-openshift-default-cf58d98f9-6gnzd"] Oct 30 13:04:02.991474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:02.991441 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:03.233290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:03.226203 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:04.002676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:04.002642 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" path="/var/lib/kubelet/pods/5e0cbb8a-8a4c-440a-923b-3aa8a52ac251/volumes" Oct 30 13:04:04.761078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:04.761046 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:05.078030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.025986 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:05.103291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.087537 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk"] Oct 30 13:04:05.103291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.088768 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" Oct 30 13:04:05.103291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.088792 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" Oct 30 13:04:05.103291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.088879 5515 cpu_manager.go:414] "containerMap: removing container" podUID="a97c75bb-9fae-4978-a460-39b24d898ad5" containerName="main" Oct 30 13:04:05.103291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.088888 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97c75bb-9fae-4978-a460-39b24d898ad5" containerName="main" Oct 30 13:04:05.103291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.089061 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97c75bb-9fae-4978-a460-39b24d898ad5" containerName="main" Oct 30 13:04:05.103291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.089076 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" Oct 30 13:04:05.149290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.113048 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk"] Oct 30 13:04:05.149290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.113178 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:05.149290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.148475 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-405eb1\"" Oct 30 13:04:05.221411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.156467 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-405eb1-dockercfg-jxvt6\"" Oct 30 13:04:05.263974 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.263923 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxf27\" (UniqueName: \"kubernetes.io/projected/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-kube-api-access-kxf27\") pod \"payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk\" (UID: \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:05.264208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.264116 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk\" (UID: \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:05.365346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.365217 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxf27\" (UniqueName: \"kubernetes.io/projected/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-kube-api-access-kxf27\") pod \"payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk\" (UID: \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:05.365346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.365340 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk\" (UID: \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:05.366076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.366057 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk\" (UID: \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:05.389434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.389402 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxf27\" (UniqueName: \"kubernetes.io/projected/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-kube-api-access-kxf27\") pod \"payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk\" (UID: \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:05.456660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.456619 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:05.764419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:05.764390 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk"] Oct 30 13:04:05.764667 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:04:05.764640 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6d5a88_2ebf_4b67_9be2_f6968b9234df.slice/crio-8897f8b73bddf62414d41fa1d010344fa68488bbe70d996b5820ae7b6ddd476b WatchSource:0}: Error finding container 8897f8b73bddf62414d41fa1d010344fa68488bbe70d996b5820ae7b6ddd476b: Status 404 returned error can't find the container with id 8897f8b73bddf62414d41fa1d010344fa68488bbe70d996b5820ae7b6ddd476b Oct 30 13:04:06.568682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.568631 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" event={"ID":"cd6d5a88-2ebf-4b67-9be2-f6968b9234df","Type":"ContainerStarted","Data":"fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6"} Oct 30 13:04:06.569135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.568709 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" event={"ID":"cd6d5a88-2ebf-4b67-9be2-f6968b9234df","Type":"ContainerStarted","Data":"8897f8b73bddf62414d41fa1d010344fa68488bbe70d996b5820ae7b6ddd476b"} Oct 30 13:04:06.569135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.568859 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:06.621287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.621219 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" podStartSLOduration=2.621202092 podStartE2EDuration="2.621202092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:06.618810375 +0000 UTC m=+1864.745611273" watchObservedRunningTime="2026-10-30 13:04:06.621202092 +0000 UTC m=+1864.748003021" Oct 30 13:04:06.679661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.679628 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:06.735100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.735066 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn"] Oct 30 13:04:06.735792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.735776 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" Oct 30 13:04:06.735852 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.735794 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" Oct 30 13:04:06.735912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.735902 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0cbb8a-8a4c-440a-923b-3aa8a52ac251" containerName="istio-proxy" Oct 30 13:04:06.738754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.738738 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:06.761074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.761047 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3a236d-dockercfg-vhmgn\"" Oct 30 13:04:06.771848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.771809 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3a236d\"" Oct 30 13:04:06.780644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.780605 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn"] Oct 30 13:04:06.784855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.784821 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88"] Oct 30 13:04:06.789322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.789293 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:06.806405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.806362 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88"] Oct 30 13:04:06.909294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.903346 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rc7w\" (UniqueName: \"kubernetes.io/projected/48b0c77f-f12b-4455-86d6-96927ed354fa-kube-api-access-2rc7w\") pod \"payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88\" (UID: \"48b0c77f-f12b-4455-86d6-96927ed354fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:06.909294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.903413 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b0c77f-f12b-4455-86d6-96927ed354fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88\" (UID: \"48b0c77f-f12b-4455-86d6-96927ed354fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:06.909294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.903464 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554adfd-5c59-4989-9061-366702999398-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn\" (UID: \"d554adfd-5c59-4989-9061-366702999398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:06.909294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:06.903552 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qghh\" (UniqueName: \"kubernetes.io/projected/d554adfd-5c59-4989-9061-366702999398-kube-api-access-4qghh\") pod \"payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn\" (UID: \"d554adfd-5c59-4989-9061-366702999398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:07.049368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.046731 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rc7w\" (UniqueName: \"kubernetes.io/projected/48b0c77f-f12b-4455-86d6-96927ed354fa-kube-api-access-2rc7w\") pod \"payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88\" (UID: \"48b0c77f-f12b-4455-86d6-96927ed354fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:07.049368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.046803 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b0c77f-f12b-4455-86d6-96927ed354fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88\" (UID: \"48b0c77f-f12b-4455-86d6-96927ed354fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:07.049368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.046867 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554adfd-5c59-4989-9061-366702999398-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn\" (UID: \"d554adfd-5c59-4989-9061-366702999398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:07.049368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.047030 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qghh\" (UniqueName: \"kubernetes.io/projected/d554adfd-5c59-4989-9061-366702999398-kube-api-access-4qghh\") pod \"payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn\" (UID: \"d554adfd-5c59-4989-9061-366702999398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:07.051127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.050131 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b0c77f-f12b-4455-86d6-96927ed354fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88\" (UID: \"48b0c77f-f12b-4455-86d6-96927ed354fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:07.051127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.051000 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554adfd-5c59-4989-9061-366702999398-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn\" (UID: \"d554adfd-5c59-4989-9061-366702999398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:07.053122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.053096 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.109896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.109861 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rc7w\" (UniqueName: \"kubernetes.io/projected/48b0c77f-f12b-4455-86d6-96927ed354fa-kube-api-access-2rc7w\") pod \"payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88\" (UID: \"48b0c77f-f12b-4455-86d6-96927ed354fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:07.115020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.114989 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:07.135482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.134004 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qghh\" (UniqueName: \"kubernetes.io/projected/d554adfd-5c59-4989-9061-366702999398-kube-api-access-4qghh\") pod \"payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn\" (UID: \"d554adfd-5c59-4989-9061-366702999398\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:07.215974 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.215939 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9"] Oct 30 13:04:07.351805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.351772 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:07.450361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.441026 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.561489 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:04:07.561449 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b0c77f_f12b_4455_86d6_96927ed354fa.slice/crio-fb279acdb41147547c780c6ab75cf681c345cd495a4c5762de675e6d45453725 WatchSource:0}: Error finding container fb279acdb41147547c780c6ab75cf681c345cd495a4c5762de675e6d45453725: Status 404 returned error can't find the container with id fb279acdb41147547c780c6ab75cf681c345cd495a4c5762de675e6d45453725 Oct 30 13:04:07.562430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.562407 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88"] Oct 30 13:04:07.578181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.578154 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" event={"ID":"48b0c77f-f12b-4455-86d6-96927ed354fa","Type":"ContainerStarted","Data":"fb279acdb41147547c780c6ab75cf681c345cd495a4c5762de675e6d45453725"} Oct 30 13:04:07.664504 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:04:07.664477 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd554adfd_5c59_4989_9061_366702999398.slice/crio-7597b988732800ddf68a90c6f80fb4f502268eb98a58b4a4167926c054a60ff5 WatchSource:0}: Error finding container 7597b988732800ddf68a90c6f80fb4f502268eb98a58b4a4167926c054a60ff5: Status 404 returned error can't find the container with id 7597b988732800ddf68a90c6f80fb4f502268eb98a58b4a4167926c054a60ff5 Oct 30 13:04:07.664825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.664802 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn"] Oct 30 13:04:07.683180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.683062 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.997538 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:07.992554 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:08.219414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:08.219382 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:08.590673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:08.590636 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" event={"ID":"d554adfd-5c59-4989-9061-366702999398","Type":"ContainerStarted","Data":"7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2"} Oct 30 13:04:08.591107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:08.590691 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" event={"ID":"d554adfd-5c59-4989-9061-366702999398","Type":"ContainerStarted","Data":"7597b988732800ddf68a90c6f80fb4f502268eb98a58b4a4167926c054a60ff5"} Oct 30 13:04:08.591107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:08.590870 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:08.592708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:08.592683 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" event={"ID":"48b0c77f-f12b-4455-86d6-96927ed354fa","Type":"ContainerStarted","Data":"e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29"} Oct 30 13:04:08.592862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:08.592783 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:08.631248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:08.631192 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" podStartSLOduration=2.63117733 podStartE2EDuration="2.63117733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:08.629939054 +0000 UTC m=+1866.756739948" watchObservedRunningTime="2026-10-30 13:04:08.63117733 +0000 UTC m=+1866.757978228" Oct 30 13:04:08.678357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:08.678308 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" podStartSLOduration=2.678291206 podStartE2EDuration="2.678291206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:08.676904361 +0000 UTC m=+1866.803705261" watchObservedRunningTime="2026-10-30 13:04:08.678291206 +0000 UTC m=+1866.805092105" Oct 30 13:04:08.848989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:08.848894 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:10.023899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:10.023860 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:11.160027 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:11.159991 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:11.419722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:11.419635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:12.217397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.217321 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" podUID="3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60" containerName="payload-pre-processing" containerID="cri-o://0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c" gracePeriod=25 Oct 30 13:04:12.530655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.530612 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:04:12.579437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.579407 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-kube-api-access-q72h8\" (UniqueName: \"kubernetes.io/projected/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-kube-api-access-q72h8\") pod \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\" (UID: \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\") " Oct 30 13:04:12.579600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.579554 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-plugins-config-volume\") pod \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\" (UID: \"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60\") " Oct 30 13:04:12.579912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.579883 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-plugins-config-volume" pod "3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60" (UID: "3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:12.581579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.581552 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-kube-api-access-q72h8" pod "3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60" (UID: "3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60"). InnerVolumeSpecName "kube-api-access-q72h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:12.643909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.643878 5515 generic.go:356] "Generic (PLEG): container finished" podID="3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60" containerID="0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c" exitCode=0 Oct 30 13:04:12.644091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.643923 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" Oct 30 13:04:12.644091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.643971 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" event={"ID":"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60","Type":"ContainerDied","Data":"0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c"} Oct 30 13:04:12.644091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.644032 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9" event={"ID":"3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60","Type":"ContainerDied","Data":"7f70c4747c371aa703f2ebec0b156d04d8feb2cb9127221977bea75d9c8a5e2d"} Oct 30 13:04:12.644091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.644078 5515 scope.go:122] "RemoveContainer" containerID="0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c" Oct 30 13:04:12.655208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.655187 5515 scope.go:122] "RemoveContainer" containerID="0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c" Oct 30 13:04:12.655548 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:12.655529 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c\": container with ID starting with 0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c not found: ID does not exist" containerID="0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c" Oct 30 13:04:12.655627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.655558 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c"} err="failed to get container status \"0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c\": rpc error: code = NotFound desc = could not find container \"0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c\": container with ID starting with 0b3c36a227bd6def6a40064fcc7399a9b91d88df1301a2a38e06bc5814c4f48c not found: ID does not exist" Oct 30 13:04:12.681356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.681334 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q72h8\" (UniqueName: \"kubernetes.io/projected/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-kube-api-access-q72h8\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:12.681356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.681353 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:12.724596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.724558 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9"] Oct 30 13:04:12.727756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.727136 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a2768f-6b7479d5c9-g7tf9"] Oct 30 13:04:12.923216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.923182 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt"] Oct 30 13:04:12.924179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.924155 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60" containerName="payload-pre-processing" Oct 30 13:04:12.924179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.924178 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60" containerName="payload-pre-processing" Oct 30 13:04:12.924405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.924332 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60" containerName="payload-pre-processing" Oct 30 13:04:12.928032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.928014 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:12.942430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.942403 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt"] Oct 30 13:04:12.944620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.944601 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a1ee32\"" Oct 30 13:04:12.974118 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.974077 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:12.985988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.985908 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjsb\" (UniqueName: \"kubernetes.io/projected/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-kube-api-access-8kjsb\") pod \"payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt\" (UID: \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:12.986141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:12.986122 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt\" (UID: \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:13.149357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:13.142005 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt\" (UID: \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:13.149357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:13.142281 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjsb\" (UniqueName: \"kubernetes.io/projected/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-kube-api-access-8kjsb\") pod \"payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt\" (UID: \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:13.196318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:13.189114 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt\" (UID: \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:13.219294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:13.217352 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:13.392381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:13.391228 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjsb\" (UniqueName: \"kubernetes.io/projected/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-kube-api-access-8kjsb\") pod \"payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt\" (UID: \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:13.543166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:13.543127 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:13.868983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:13.868950 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt"] Oct 30 13:04:14.026343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:14.026304 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60" path="/var/lib/kubelet/pods/3ff9dcd0-8e60-4bd3-b0a1-abc4a0146e60/volumes" Oct 30 13:04:14.399294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:14.397865 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:14.665002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:14.664904 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" event={"ID":"9d9a314f-a99e-4e31-8adf-aebc11adf7fc","Type":"ContainerStarted","Data":"812660ba763aa3124213f3360b118088131cc06e432101ed28e38b03899349e6"} Oct 30 13:04:14.665002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:14.664960 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" event={"ID":"9d9a314f-a99e-4e31-8adf-aebc11adf7fc","Type":"ContainerStarted","Data":"e6000a6c9c91070362c5f699da867bf65c1088c010c5046dff8715121ca91f23"} Oct 30 13:04:14.665002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:14.664987 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:14.712869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:14.712815 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" podStartSLOduration=2.712799982 podStartE2EDuration="2.712799982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:14.711608525 +0000 UTC m=+1872.838409424" watchObservedRunningTime="2026-10-30 13:04:14.712799982 +0000 UTC m=+1872.839600880" Oct 30 13:04:15.957598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:15.957549 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:17.389348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:17.383685 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:17.589387 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:17.585095 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:17.609896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:17.609859 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:18.700853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:18.700819 5515 generic.go:356] "Generic (PLEG): container finished" podID="03e78981-662f-45c9-b066-e6cee6e117e8" containerID="cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838" exitCode=0 Oct 30 13:04:18.701343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:18.700884 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" event={"ID":"03e78981-662f-45c9-b066-e6cee6e117e8","Type":"ContainerDied","Data":"cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838"} Oct 30 13:04:18.701400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:18.701379 5515 scope.go:122] "RemoveContainer" containerID="cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838" Oct 30 13:04:18.703114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:18.703077 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:04:18.703205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:18.703153 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:04:18.703205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:18.703191 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:04:19.145498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:19.145465 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:19.626834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:19.626758 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:04:19.626987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:19.626836 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:04:19.741287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:19.737782 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" event={"ID":"03e78981-662f-45c9-b066-e6cee6e117e8","Type":"ContainerStarted","Data":"9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494"} Oct 30 13:04:19.956925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:19.956879 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:04:19.958296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:19.958243 5515 patch_prober.go:29] interesting pod/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:04:19.958435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:19.958316 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 13:04:20.672830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.672794 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq"] Oct 30 13:04:20.677015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.676993 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.734611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.734577 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3a236d-serving-cert\"" Oct 30 13:04:20.752219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.752179 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq"] Oct 30 13:04:20.753703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.753561 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.753703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.753640 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.753703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.753673 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-metrics-certs\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.753938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.753743 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlb6f\" (UniqueName: \"kubernetes.io/projected/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-kube-api-access-mlb6f\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.868442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.868402 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.886425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.886387 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.886668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.886651 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-metrics-certs\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.899460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.899422 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlb6f\" (UniqueName: \"kubernetes.io/projected/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-kube-api-access-mlb6f\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.899882 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:20.890438 5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-serving-cert: secret "maas-api-e2e-api-a-3a236d-serving-cert" not found Oct 30 13:04:20.900122 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:20.900104 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls podName:d16d4d48-0912-4b0a-ac44-7d3e6b2b298a nodeName:}" failed. No retries permitted until 2026-10-30 13:04:21.400079263 +0000 UTC m=+1879.526880138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls") pod "maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" (UID: "d16d4d48-0912-4b0a-ac44-7d3e6b2b298a") : secret "maas-api-e2e-api-a-3a236d-serving-cert" not found Oct 30 13:04:20.900452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.869740 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.913417 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.904070 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:20.923549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.919724 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-metrics-certs\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.933932 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.933862 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlb6f\" (UniqueName: \"kubernetes.io/projected/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-kube-api-access-mlb6f\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:20.968887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.968851 5515 patch_prober.go:29] interesting pod/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:04:20.969150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:20.969119 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 13:04:21.136408 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:21.136372 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:21.439290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:21.434454 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:21.459608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:21.448939 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls\") pod \"maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:21.590545 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:21.590509 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:21.964145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:21.960310 5515 patch_prober.go:29] interesting pod/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:04:21.964145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:21.960373 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 13:04:22.167011 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:04:22.166980 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16d4d48_0912_4b0a_ac44_7d3e6b2b298a.slice/crio-5a3c37c79956fcd8af74fe4bc527f40af43abd315fa886df1a564482773b2f9e WatchSource:0}: Error finding container 5a3c37c79956fcd8af74fe4bc527f40af43abd315fa886df1a564482773b2f9e: Status 404 returned error can't find the container with id 5a3c37c79956fcd8af74fe4bc527f40af43abd315fa886df1a564482773b2f9e Oct 30 13:04:22.168612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:22.168584 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq"] Oct 30 13:04:22.482506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:22.482474 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:22.926135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:22.926100 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" event={"ID":"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a","Type":"ContainerStarted","Data":"1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52"} Oct 30 13:04:22.926364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:22.926159 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" event={"ID":"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a","Type":"ContainerStarted","Data":"5a3c37c79956fcd8af74fe4bc527f40af43abd315fa886df1a564482773b2f9e"} Oct 30 13:04:22.926364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:22.926225 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:22.957037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:22.956999 5515 patch_prober.go:29] interesting pod/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:04:22.957216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:22.957060 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 13:04:22.971540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:22.971486 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" podStartSLOduration=2.461652919 podStartE2EDuration="2.971470429s" totalImagesPullingTime="509.81751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:22.968180634 +0000 UTC m=+1881.094981533" watchObservedRunningTime="2026-10-30 13:04:22.971470429 +0000 UTC m=+1881.098271326" Oct 30 13:04:22.972407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:22.972291 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:04:23.956670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:23.956638 5515 patch_prober.go:29] interesting pod/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:04:23.956859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:23.956691 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 13:04:24.129293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:24.122207 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:24.956936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:24.956894 5515 patch_prober.go:29] interesting pod/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 13:04:24.957122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:24.956976 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 13:04:25.676799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:25.676771 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:04:25.960956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:25.960876 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:04:25.961879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:25.961861 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:04:26.275512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:26.275436 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:27.968925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:27.968889 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk"] Oct 30 13:04:27.972351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:27.972335 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:27.990250 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:27.990214 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-metrics-certs\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:27.990716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:27.990688 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bf50b1-2822-4430-8305-a159a5a22912-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:27.990991 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:27.990972 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfr6s\" (UniqueName: \"kubernetes.io/projected/19bf50b1-2822-4430-8305-a159a5a22912-kube-api-access-dfr6s\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:27.991180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:27.991161 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:27.994349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:27.994325 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a1ee32-serving-cert\"" Oct 30 13:04:28.008077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.008039 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk"] Oct 30 13:04:28.092614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.092579 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfr6s\" (UniqueName: \"kubernetes.io/projected/19bf50b1-2822-4430-8305-a159a5a22912-kube-api-access-dfr6s\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:28.092801 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.092637 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:28.092801 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.092734 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-metrics-certs\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:28.092923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.092908 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bf50b1-2822-4430-8305-a159a5a22912-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:28.093542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.093521 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bf50b1-2822-4430-8305-a159a5a22912-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:28.095017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.094996 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:28.095155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.095134 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-metrics-certs\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:28.115841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.115809 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfr6s\" (UniqueName: \"kubernetes.io/projected/19bf50b1-2822-4430-8305-a159a5a22912-kube-api-access-dfr6s\") pod \"maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:28.164195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.164162 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:28.292342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.291887 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:28.384458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.384419 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:28.633151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.633115 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk"] Oct 30 13:04:28.633430 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:04:28.633404 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bf50b1_2822_4430_8305_a159a5a22912.slice/crio-20a721fb7af7f8612c1c2232bf60d602717a65253c5abec066701dc0a5065a45 WatchSource:0}: Error finding container 20a721fb7af7f8612c1c2232bf60d602717a65253c5abec066701dc0a5065a45: Status 404 returned error can't find the container with id 20a721fb7af7f8612c1c2232bf60d602717a65253c5abec066701dc0a5065a45 Oct 30 13:04:28.939620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.939593 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:04:28.972175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:28.972095 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" event={"ID":"19bf50b1-2822-4430-8305-a159a5a22912","Type":"ContainerStarted","Data":"20a721fb7af7f8612c1c2232bf60d602717a65253c5abec066701dc0a5065a45"} Oct 30 13:04:28.989791 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:28.989760 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:04:30.032365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:30.032173 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:30.053381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:30.053347 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:30.056569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:30.056525 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" event={"ID":"19bf50b1-2822-4430-8305-a159a5a22912","Type":"ContainerStarted","Data":"fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846"} Oct 30 13:04:30.098349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:30.098288 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" podStartSLOduration=2.774027846 podStartE2EDuration="3.098258202s" totalImagesPullingTime="324.230356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:30.094330307 +0000 UTC m=+1888.221131205" watchObservedRunningTime="2026-10-30 13:04:30.098258202 +0000 UTC m=+1888.225059125" Oct 30 13:04:30.928796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:30.928759 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:04:32.012721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:32.012654 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:34.013729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:34.013689 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:35.843924 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:35.843886 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:36.096704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:36.096622 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:04:37.548040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:37.548005 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:39.067403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:39.067362 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:39.743292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:39.738336 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:40.022940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:40.022861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:40.278905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:40.278825 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z"] Oct 30 13:04:40.279351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:40.279040 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" containerID="cri-o://902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db" gracePeriod=30 Oct 30 13:04:40.462881 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:40.462839 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:41.887869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:41.887829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:43.619376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:43.619333 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk"] Oct 30 13:04:43.877981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:43.877894 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:44.117291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:44.116651 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:45.494237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.494194 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:45.629923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.629895 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 13:04:45.675870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.675830 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-podinfo\") pod \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " Oct 30 13:04:45.676087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.675905 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-certs\") pod \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " Oct 30 13:04:45.676087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.675961 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-socket\") pod \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " Oct 30 13:04:45.676087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.675979 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istiod-ca-cert\") pod \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " Oct 30 13:04:45.676087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676012 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-kube-api-access-h6t4g\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-kube-api-access-h6t4g\") pod \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " Oct 30 13:04:45.676345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676217 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-envoy\") pod \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " Oct 30 13:04:45.676345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676277 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-socket" pod "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" (UID: "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:45.676345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676294 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-certs" pod "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" (UID: "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:45.676503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676317 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-token\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-token\") pod \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " Oct 30 13:04:45.676503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676433 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-data\") pod \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " Oct 30 13:04:45.676601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676503 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istiod-ca-cert" pod "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" (UID: "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:45.676601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676521 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-credential-socket\") pod \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\" (UID: \"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c\") " Oct 30 13:04:45.676809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676787 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-data" pod "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" (UID: "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:45.676904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.676879 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-credential-socket" pod "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" (UID: "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:45.677452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.677425 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.677578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.677453 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.677578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.677470 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.677578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.677485 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.677578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.677499 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.678670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.678634 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-kube-api-access-h6t4g" pod "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" (UID: "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c"). InnerVolumeSpecName "kube-api-access-h6t4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:45.678933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.678908 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-envoy" pod "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" (UID: "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:45.679159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.679141 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-token" pod "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" (UID: "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:45.679645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.679629 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-podinfo" pod "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" (UID: "0df1bf5b-456f-4aea-bd62-c11eaa4ad45c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:45.778800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.778758 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.778800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.778801 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.779013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.778815 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.779013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.778829 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6t4g\" (UniqueName: \"kubernetes.io/projected/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c-kube-api-access-h6t4g\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:45.802920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:45.802880 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:46.209865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.209821 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:46.347034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.346941 5515 generic.go:356] "Generic (PLEG): container finished" podID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerID="902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db" exitCode=0 Oct 30 13:04:46.347034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.346998 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" Oct 30 13:04:46.347034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.347019 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" event={"ID":"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c","Type":"ContainerDied","Data":"902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db"} Oct 30 13:04:46.347378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.347074 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z" event={"ID":"0df1bf5b-456f-4aea-bd62-c11eaa4ad45c","Type":"ContainerDied","Data":"1161cf3e9e14cc86620f0984ee04ce8bedd1a7d0ff273dea1d1fd4698bc2df2e"} Oct 30 13:04:46.347378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.347096 5515 scope.go:122] "RemoveContainer" containerID="902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db" Oct 30 13:04:46.357552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.357531 5515 scope.go:122] "RemoveContainer" containerID="902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db" Oct 30 13:04:46.357905 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:46.357883 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db\": container with ID starting with 902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db not found: ID does not exist" containerID="902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db" Oct 30 13:04:46.357964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.357911 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db"} err="failed to get container status \"902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db\": rpc error: code = NotFound desc = could not find container \"902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db\": container with ID starting with 902285a2408d0e97e12f6666332236c00819beefd3d78f2369a4ae0c200ea8db not found: ID does not exist" Oct 30 13:04:46.439279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.439238 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z"] Oct 30 13:04:46.454088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.454053 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rkp5z"] Oct 30 13:04:46.529058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.529016 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:46.990201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:46.990164 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:47.404456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:47.404421 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:48.003017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:48.002982 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" path="/var/lib/kubelet/pods/0df1bf5b-456f-4aea-bd62-c11eaa4ad45c/volumes" Oct 30 13:04:48.289351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:48.288428 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:48.511220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:48.511185 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh"] Oct 30 13:04:48.511489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:48.511463 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" containerID="cri-o://9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494" gracePeriod=30 Oct 30 13:04:48.533015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:48.532980 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:48.620446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:48.620340 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" podUID="cd6d5a88-2ebf-4b67-9be2-f6968b9234df" containerName="payload-processing" containerID="cri-o://fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6" gracePeriod=25 Oct 30 13:04:48.770294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:48.765482 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:49.019101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.019069 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:49.102329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.099629 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-plugins-config-volume\") pod \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\" (UID: \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\") " Oct 30 13:04:49.102329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.099767 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-kube-api-access-kxf27\" (UniqueName: \"kubernetes.io/projected/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-kube-api-access-kxf27\") pod \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\" (UID: \"cd6d5a88-2ebf-4b67-9be2-f6968b9234df\") " Oct 30 13:04:49.102329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.101686 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-plugins-config-volume" pod "cd6d5a88-2ebf-4b67-9be2-f6968b9234df" (UID: "cd6d5a88-2ebf-4b67-9be2-f6968b9234df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:49.119312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.113220 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-kube-api-access-kxf27" pod "cd6d5a88-2ebf-4b67-9be2-f6968b9234df" (UID: "cd6d5a88-2ebf-4b67-9be2-f6968b9234df"). InnerVolumeSpecName "kube-api-access-kxf27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:49.147511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.147209 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:49.202429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.202344 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:49.202429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.202373 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxf27\" (UniqueName: \"kubernetes.io/projected/cd6d5a88-2ebf-4b67-9be2-f6968b9234df-kube-api-access-kxf27\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:49.370989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.370950 5515 generic.go:356] "Generic (PLEG): container finished" podID="cd6d5a88-2ebf-4b67-9be2-f6968b9234df" containerID="fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6" exitCode=0 Oct 30 13:04:49.371213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.370999 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" Oct 30 13:04:49.371213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.371035 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" event={"ID":"cd6d5a88-2ebf-4b67-9be2-f6968b9234df","Type":"ContainerDied","Data":"fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6"} Oct 30 13:04:49.371213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.371095 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk" event={"ID":"cd6d5a88-2ebf-4b67-9be2-f6968b9234df","Type":"ContainerDied","Data":"8897f8b73bddf62414d41fa1d010344fa68488bbe70d996b5820ae7b6ddd476b"} Oct 30 13:04:49.371213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.371125 5515 scope.go:122] "RemoveContainer" containerID="fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6" Oct 30 13:04:49.383521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.383498 5515 scope.go:122] "RemoveContainer" containerID="fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6" Oct 30 13:04:49.383852 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:49.383826 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6\": container with ID starting with fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6 not found: ID does not exist" containerID="fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6" Oct 30 13:04:49.383961 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.383859 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6"} err="failed to get container status \"fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6\": rpc error: code = NotFound desc = could not find container \"fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6\": container with ID starting with fdab36f010ea3450bba9c68af11a15820cbb4214b06d20f3dcbf18201a3bfaa6 not found: ID does not exist" Oct 30 13:04:49.443808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.443771 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk"] Oct 30 13:04:49.470436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.470363 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-78879888cb-zrrpk"] Oct 30 13:04:49.752513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:49.752429 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:50.002963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:50.002885 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6d5a88-2ebf-4b67-9be2-f6968b9234df" path="/var/lib/kubelet/pods/cd6d5a88-2ebf-4b67-9be2-f6968b9234df/volumes" Oct 30 13:04:50.466011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:50.465955 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:50.714863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:50.714825 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:50.986327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:50.986261 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:52.388500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.388463 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:52.646708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.646631 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:52.718591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.718556 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d"] Oct 30 13:04:52.718816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.718781 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" containerID="cri-o://cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf" gracePeriod=30 Oct 30 13:04:52.731008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.730978 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz"] Oct 30 13:04:52.731188 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.731150 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" containerID="cri-o://34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22" gracePeriod=30 Oct 30 13:04:52.733855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.733831 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4"] Oct 30 13:04:52.734079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.734057 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" containerID="cri-o://c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a" gracePeriod=30 Oct 30 13:04:52.742868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.742841 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd"] Oct 30 13:04:52.743028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.743008 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" containerID="cri-o://d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5" gracePeriod=30 Oct 30 13:04:52.909101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:52.909004 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:53.166420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.165135 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:53.789391 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.789366 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:04:53.833699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.833608 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-certs\") pod \"03e78981-662f-45c9-b066-e6cee6e117e8\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " Oct 30 13:04:53.833699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.833668 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03e78981-662f-45c9-b066-e6cee6e117e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e78981-662f-45c9-b066-e6cee6e117e8-istio-podinfo\") pod \"03e78981-662f-45c9-b066-e6cee6e117e8\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " Oct 30 13:04:53.833699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.833694 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-kube-api-access-dkd9x\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-kube-api-access-dkd9x\") pod \"03e78981-662f-45c9-b066-e6cee6e117e8\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " Oct 30 13:04:53.833987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.833722 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-envoy\") pod \"03e78981-662f-45c9-b066-e6cee6e117e8\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " Oct 30 13:04:53.833987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.833792 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-socket\") pod \"03e78981-662f-45c9-b066-e6cee6e117e8\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " Oct 30 13:04:53.833987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.833836 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e78981-662f-45c9-b066-e6cee6e117e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e78981-662f-45c9-b066-e6cee6e117e8-istiod-ca-cert\") pod \"03e78981-662f-45c9-b066-e6cee6e117e8\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " Oct 30 13:04:53.833987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.833876 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-data\") pod \"03e78981-662f-45c9-b066-e6cee6e117e8\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " Oct 30 13:04:53.833987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.833942 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-certs" pod "03e78981-662f-45c9-b066-e6cee6e117e8" (UID: "03e78981-662f-45c9-b066-e6cee6e117e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:53.834230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834043 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-credential-socket\") pod \"03e78981-662f-45c9-b066-e6cee6e117e8\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " Oct 30 13:04:53.834230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834073 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-istio-token\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-istio-token\") pod \"03e78981-662f-45c9-b066-e6cee6e117e8\" (UID: \"03e78981-662f-45c9-b066-e6cee6e117e8\") " Oct 30 13:04:53.834230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834170 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-socket" pod "03e78981-662f-45c9-b066-e6cee6e117e8" (UID: "03e78981-662f-45c9-b066-e6cee6e117e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:53.834399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834322 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-data" pod "03e78981-662f-45c9-b066-e6cee6e117e8" (UID: "03e78981-662f-45c9-b066-e6cee6e117e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:53.834451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834400 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-credential-socket" pod "03e78981-662f-45c9-b066-e6cee6e117e8" (UID: "03e78981-662f-45c9-b066-e6cee6e117e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:53.834451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834419 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e78981-662f-45c9-b066-e6cee6e117e8-istiod-ca-cert" pod "03e78981-662f-45c9-b066-e6cee6e117e8" (UID: "03e78981-662f-45c9-b066-e6cee6e117e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:53.834933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834906 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.834933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834933 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.835136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834948 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e78981-662f-45c9-b066-e6cee6e117e8-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.835136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834964 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.835136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.834976 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.836342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.836240 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-kube-api-access-dkd9x" pod "03e78981-662f-45c9-b066-e6cee6e117e8" (UID: "03e78981-662f-45c9-b066-e6cee6e117e8"). InnerVolumeSpecName "kube-api-access-dkd9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:53.836342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.836302 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-envoy" pod "03e78981-662f-45c9-b066-e6cee6e117e8" (UID: "03e78981-662f-45c9-b066-e6cee6e117e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:53.836472 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.836433 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03e78981-662f-45c9-b066-e6cee6e117e8-istio-podinfo" pod "03e78981-662f-45c9-b066-e6cee6e117e8" (UID: "03e78981-662f-45c9-b066-e6cee6e117e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:53.836472 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.836463 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-istio-token" pod "03e78981-662f-45c9-b066-e6cee6e117e8" (UID: "03e78981-662f-45c9-b066-e6cee6e117e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:53.936950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.936906 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e78981-662f-45c9-b066-e6cee6e117e8-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.936950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.936945 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkd9x\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-kube-api-access-dkd9x\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.937198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.936960 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e78981-662f-45c9-b066-e6cee6e117e8-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:53.937198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:53.936975 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e78981-662f-45c9-b066-e6cee6e117e8-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:54.194296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.192898 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:54.427821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.421364 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:54.499442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.499350 5515 generic.go:356] "Generic (PLEG): container finished" podID="03e78981-662f-45c9-b066-e6cee6e117e8" containerID="9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494" exitCode=0 Oct 30 13:04:54.499442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.499395 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" event={"ID":"03e78981-662f-45c9-b066-e6cee6e117e8","Type":"ContainerDied","Data":"9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494"} Oct 30 13:04:54.499442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.499420 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" Oct 30 13:04:54.499442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.499439 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh" event={"ID":"03e78981-662f-45c9-b066-e6cee6e117e8","Type":"ContainerDied","Data":"36ea246f070e505a7d173114450effbd2a16403b18bf1a15ec13dead7fc4006a"} Oct 30 13:04:54.499753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.499466 5515 scope.go:122] "RemoveContainer" containerID="9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494" Oct 30 13:04:54.510317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.510298 5515 scope.go:122] "RemoveContainer" containerID="cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838" Oct 30 13:04:54.521231 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.521205 5515 scope.go:122] "RemoveContainer" containerID="9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494" Oct 30 13:04:54.521594 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:54.521573 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494\": container with ID starting with 9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494 not found: ID does not exist" containerID="9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494" Oct 30 13:04:54.521663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.521602 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494"} err="failed to get container status \"9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494\": rpc error: code = NotFound desc = could not find container \"9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494\": container with ID starting with 9669bd2e2bf9853d655b59749904985151d70f4c09e67a74b2bfb35cf13a4494 not found: ID does not exist" Oct 30 13:04:54.521663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.521622 5515 scope.go:122] "RemoveContainer" containerID="cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838" Oct 30 13:04:54.521904 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:54.521882 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838\": container with ID starting with cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838 not found: ID does not exist" containerID="cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838" Oct 30 13:04:54.521961 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.521915 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838"} err="failed to get container status \"cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838\": rpc error: code = NotFound desc = could not find container \"cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838\": container with ID starting with cf9b5e42174baa7587393344ab918715f887debe285c29cf0fec48c02c441838 not found: ID does not exist" Oct 30 13:04:54.618350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.615648 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh"] Oct 30 13:04:54.618350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.615715 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a2768f-openshift-default-7f8c86cc95-bjzlh"] Oct 30 13:04:54.662360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.661243 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:54.904674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:54.904634 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:56.003512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.003480 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" path="/var/lib/kubelet/pods/03e78981-662f-45c9-b066-e6cee6e117e8/volumes" Oct 30 13:04:56.608669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.608623 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:56.715646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.715603 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8"] Oct 30 13:04:56.716343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716324 5515 cpu_manager.go:414] "containerMap: removing container" podUID="cd6d5a88-2ebf-4b67-9be2-f6968b9234df" containerName="payload-processing" Oct 30 13:04:56.716343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716343 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6d5a88-2ebf-4b67-9be2-f6968b9234df" containerName="payload-processing" Oct 30 13:04:56.716577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716369 5515 cpu_manager.go:414] "containerMap: removing container" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" Oct 30 13:04:56.716577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716376 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" Oct 30 13:04:56.716577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716392 5515 cpu_manager.go:414] "containerMap: removing container" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" Oct 30 13:04:56.716577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716400 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" Oct 30 13:04:56.716577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716431 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" Oct 30 13:04:56.716577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716439 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" Oct 30 13:04:56.716857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716625 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6d5a88-2ebf-4b67-9be2-f6968b9234df" containerName="payload-processing" Oct 30 13:04:56.716857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716643 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" Oct 30 13:04:56.716857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716655 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e78981-662f-45c9-b066-e6cee6e117e8" containerName="istio-proxy" Oct 30 13:04:56.716857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.716670 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df1bf5b-456f-4aea-bd62-c11eaa4ad45c" containerName="istio-proxy" Oct 30 13:04:56.720508 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.720486 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:04:56.741883 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.741848 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8"] Oct 30 13:04:56.749996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.749960 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fa855c\"" Oct 30 13:04:56.750183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.750048 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fa855c-dockercfg-79pm6\"" Oct 30 13:04:56.826747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.826706 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn"] Oct 30 13:04:56.835465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.829766 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6t6\" (UniqueName: \"kubernetes.io/projected/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-kube-api-access-rn6t6\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8\" (UID: \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:04:56.835676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.835571 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8\" (UID: \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:04:56.854293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.849910 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:04:56.917292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.916432 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn"] Oct 30 13:04:56.949293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.945035 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:56.949293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.946140 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8\" (UID: \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:04:56.949293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.946212 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkh9\" (UniqueName: \"kubernetes.io/projected/d8e90e60-fab3-49e7-9541-6c03658a7114-kube-api-access-fqkh9\") pod \"payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn\" (UID: \"d8e90e60-fab3-49e7-9541-6c03658a7114\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:04:56.954301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.951679 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6t6\" (UniqueName: \"kubernetes.io/projected/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-kube-api-access-rn6t6\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8\" (UID: \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:04:56.954301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.951753 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e90e60-fab3-49e7-9541-6c03658a7114-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn\" (UID: \"d8e90e60-fab3-49e7-9541-6c03658a7114\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:04:56.954301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.951772 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8\" (UID: \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:04:56.991298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:56.980794 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6t6\" (UniqueName: \"kubernetes.io/projected/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-kube-api-access-rn6t6\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8\" (UID: \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:04:57.037575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.035554 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:04:57.060655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.056544 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e90e60-fab3-49e7-9541-6c03658a7114-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn\" (UID: \"d8e90e60-fab3-49e7-9541-6c03658a7114\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:04:57.060655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.056666 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkh9\" (UniqueName: \"kubernetes.io/projected/d8e90e60-fab3-49e7-9541-6c03658a7114-kube-api-access-fqkh9\") pod \"payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn\" (UID: \"d8e90e60-fab3-49e7-9541-6c03658a7114\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:04:57.060655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.058233 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e90e60-fab3-49e7-9541-6c03658a7114-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn\" (UID: \"d8e90e60-fab3-49e7-9541-6c03658a7114\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:04:57.134784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.134744 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkh9\" (UniqueName: \"kubernetes.io/projected/d8e90e60-fab3-49e7-9541-6c03658a7114-kube-api-access-fqkh9\") pod \"payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn\" (UID: \"d8e90e60-fab3-49e7-9541-6c03658a7114\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:04:57.145036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.142826 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt"] Oct 30 13:04:57.312099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.312062 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:04:57.410323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.409716 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:57.473350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.473244 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8"] Oct 30 13:04:57.526583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.526540 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" event={"ID":"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da","Type":"ContainerStarted","Data":"7453ddd848c1d82688822fcc871c37f8da21258159abfccc612668adfe9b6f72"} Oct 30 13:04:57.692849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.692789 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:57.743032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:57.742998 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn"] Oct 30 13:04:57.765226 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:04:57.765193 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e90e60_fab3_49e7_9541_6c03658a7114.slice/crio-363cd9db373884fe674ac46e3ad4fff09456dafa17a28f53225ffe905cce18bc WatchSource:0}: Error finding container 363cd9db373884fe674ac46e3ad4fff09456dafa17a28f53225ffe905cce18bc: Status 404 returned error can't find the container with id 363cd9db373884fe674ac46e3ad4fff09456dafa17a28f53225ffe905cce18bc Oct 30 13:04:58.005511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.005468 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:58.269920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.269891 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 13:04:58.275904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.275871 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 13:04:58.353788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.353761 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 13:04:58.357428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.357400 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 13:04:58.399067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.399020 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-token\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-token\") pod \"25132bf1-77b7-4acb-940e-d36e07a12a76\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " Oct 30 13:04:58.399255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.399086 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-data\") pod \"fe03a9a4-eae9-418e-b91c-73a175fc4804\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " Oct 30 13:04:58.399255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.399136 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-token\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-token\") pod \"fe03a9a4-eae9-418e-b91c-73a175fc4804\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " Oct 30 13:04:58.399255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.399205 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-kube-api-access-frrxc\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-kube-api-access-frrxc\") pod \"25132bf1-77b7-4acb-940e-d36e07a12a76\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " Oct 30 13:04:58.399516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.399462 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-data" pod "fe03a9a4-eae9-418e-b91c-73a175fc4804" (UID: "fe03a9a4-eae9-418e-b91c-73a175fc4804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.399727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.399705 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-data\") pod \"25132bf1-77b7-4acb-940e-d36e07a12a76\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " Oct 30 13:04:58.399827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.399748 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe03a9a4-eae9-418e-b91c-73a175fc4804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe03a9a4-eae9-418e-b91c-73a175fc4804-istiod-ca-cert\") pod \"fe03a9a4-eae9-418e-b91c-73a175fc4804\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " Oct 30 13:04:58.400220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400194 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-data" pod "25132bf1-77b7-4acb-940e-d36e07a12a76" (UID: "25132bf1-77b7-4acb-940e-d36e07a12a76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.400397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400239 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-envoy\") pod \"25132bf1-77b7-4acb-940e-d36e07a12a76\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " Oct 30 13:04:58.400397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400393 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-podinfo\") pod \"fe03a9a4-eae9-418e-b91c-73a175fc4804\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " Oct 30 13:04:58.400567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400423 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-socket\") pod \"fe03a9a4-eae9-418e-b91c-73a175fc4804\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " Oct 30 13:04:58.400567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400459 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-credential-socket\") pod \"fe03a9a4-eae9-418e-b91c-73a175fc4804\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " Oct 30 13:04:58.400567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400501 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25132bf1-77b7-4acb-940e-d36e07a12a76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25132bf1-77b7-4acb-940e-d36e07a12a76-istiod-ca-cert\") pod \"25132bf1-77b7-4acb-940e-d36e07a12a76\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " Oct 30 13:04:58.400567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400561 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-certs\") pod \"25132bf1-77b7-4acb-940e-d36e07a12a76\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " Oct 30 13:04:58.400846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400605 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-envoy\") pod \"fe03a9a4-eae9-418e-b91c-73a175fc4804\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " Oct 30 13:04:58.400846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400642 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-certs\") pod \"fe03a9a4-eae9-418e-b91c-73a175fc4804\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " Oct 30 13:04:58.400846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400693 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-socket\") pod \"25132bf1-77b7-4acb-940e-d36e07a12a76\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " Oct 30 13:04:58.400846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400724 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-kube-api-access-44trj\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-kube-api-access-44trj\") pod \"fe03a9a4-eae9-418e-b91c-73a175fc4804\" (UID: \"fe03a9a4-eae9-418e-b91c-73a175fc4804\") " Oct 30 13:04:58.400846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.400805 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-podinfo\") pod \"25132bf1-77b7-4acb-940e-d36e07a12a76\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " Oct 30 13:04:58.401206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.401143 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-certs" pod "25132bf1-77b7-4acb-940e-d36e07a12a76" (UID: "25132bf1-77b7-4acb-940e-d36e07a12a76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.401206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.401185 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-certs" pod "fe03a9a4-eae9-418e-b91c-73a175fc4804" (UID: "fe03a9a4-eae9-418e-b91c-73a175fc4804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.401513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.401489 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe03a9a4-eae9-418e-b91c-73a175fc4804-istiod-ca-cert" pod "fe03a9a4-eae9-418e-b91c-73a175fc4804" (UID: "fe03a9a4-eae9-418e-b91c-73a175fc4804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:58.401938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.401912 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-credential-socket" pod "fe03a9a4-eae9-418e-b91c-73a175fc4804" (UID: "fe03a9a4-eae9-418e-b91c-73a175fc4804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.402146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.402112 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-socket" pod "fe03a9a4-eae9-418e-b91c-73a175fc4804" (UID: "fe03a9a4-eae9-418e-b91c-73a175fc4804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.402254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.402230 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-token" pod "25132bf1-77b7-4acb-940e-d36e07a12a76" (UID: "25132bf1-77b7-4acb-940e-d36e07a12a76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.402519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.402492 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-socket" pod "25132bf1-77b7-4acb-940e-d36e07a12a76" (UID: "25132bf1-77b7-4acb-940e-d36e07a12a76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.402827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.402798 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25132bf1-77b7-4acb-940e-d36e07a12a76-istiod-ca-cert" pod "25132bf1-77b7-4acb-940e-d36e07a12a76" (UID: "25132bf1-77b7-4acb-940e-d36e07a12a76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:58.402922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.402900 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-credential-socket\") pod \"25132bf1-77b7-4acb-940e-d36e07a12a76\" (UID: \"25132bf1-77b7-4acb-940e-d36e07a12a76\") " Oct 30 13:04:58.403817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.403791 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-envoy" pod "25132bf1-77b7-4acb-940e-d36e07a12a76" (UID: "25132bf1-77b7-4acb-940e-d36e07a12a76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.403971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.403844 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-envoy" pod "fe03a9a4-eae9-418e-b91c-73a175fc4804" (UID: "fe03a9a4-eae9-418e-b91c-73a175fc4804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.404336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404315 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404340 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404354 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404366 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404379 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404393 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404405 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe03a9a4-eae9-418e-b91c-73a175fc4804-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404417 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404429 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404441 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe03a9a4-eae9-418e-b91c-73a175fc4804-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404373 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-credential-socket" pod "25132bf1-77b7-4acb-940e-d36e07a12a76" (UID: "25132bf1-77b7-4acb-940e-d36e07a12a76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.404467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404455 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25132bf1-77b7-4acb-940e-d36e07a12a76-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404489 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.404888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404494 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-kube-api-access-44trj" pod "fe03a9a4-eae9-418e-b91c-73a175fc4804" (UID: "fe03a9a4-eae9-418e-b91c-73a175fc4804"). InnerVolumeSpecName "kube-api-access-44trj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.404888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.404592 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-kube-api-access-frrxc" pod "25132bf1-77b7-4acb-940e-d36e07a12a76" (UID: "25132bf1-77b7-4acb-940e-d36e07a12a76"). InnerVolumeSpecName "kube-api-access-frrxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.405079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.405060 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-podinfo" pod "fe03a9a4-eae9-418e-b91c-73a175fc4804" (UID: "fe03a9a4-eae9-418e-b91c-73a175fc4804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:58.405153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.405135 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-podinfo" pod "25132bf1-77b7-4acb-940e-d36e07a12a76" (UID: "25132bf1-77b7-4acb-940e-d36e07a12a76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:58.406139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.406118 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-token" pod "fe03a9a4-eae9-418e-b91c-73a175fc4804" (UID: "fe03a9a4-eae9-418e-b91c-73a175fc4804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.505878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.505832 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-kube-api-access-g59mq\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-kube-api-access-g59mq\") pod \"52c06c8e-6813-483b-9ea1-de079980854a\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " Oct 30 13:04:58.506076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.505986 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-istio-token\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-istio-token\") pod \"52c06c8e-6813-483b-9ea1-de079980854a\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " Oct 30 13:04:58.506076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506023 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c06c8e-6813-483b-9ea1-de079980854a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c06c8e-6813-483b-9ea1-de079980854a-istiod-ca-cert\") pod \"52c06c8e-6813-483b-9ea1-de079980854a\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " Oct 30 13:04:58.506178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506085 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-envoy\") pod \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " Oct 30 13:04:58.506178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506118 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-certs\") pod \"52c06c8e-6813-483b-9ea1-de079980854a\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " Oct 30 13:04:58.506178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506152 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-data\") pod \"52c06c8e-6813-483b-9ea1-de079980854a\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " Oct 30 13:04:58.506315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506197 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-kube-api-access-lk8mt\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-kube-api-access-lk8mt\") pod \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " Oct 30 13:04:58.506315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506234 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-socket\") pod \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " Oct 30 13:04:58.506377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506344 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-socket\") pod \"52c06c8e-6813-483b-9ea1-de079980854a\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " Oct 30 13:04:58.506421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506399 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-certs\") pod \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " Oct 30 13:04:58.506480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506465 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-data\") pod \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " Oct 30 13:04:58.506559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506545 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-envoy\") pod \"52c06c8e-6813-483b-9ea1-de079980854a\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " Oct 30 13:04:58.506598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506583 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-podinfo\") pod \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " Oct 30 13:04:58.506653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506640 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52c06c8e-6813-483b-9ea1-de079980854a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c06c8e-6813-483b-9ea1-de079980854a-istio-podinfo\") pod \"52c06c8e-6813-483b-9ea1-de079980854a\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " Oct 30 13:04:58.506687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506673 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istiod-ca-cert\") pod \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " Oct 30 13:04:58.506727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506705 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-credential-socket\") pod \"52c06c8e-6813-483b-9ea1-de079980854a\" (UID: \"52c06c8e-6813-483b-9ea1-de079980854a\") " Oct 30 13:04:58.506762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506748 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-credential-socket\") pod \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " Oct 30 13:04:58.506850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.506836 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-token\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-token\") pod \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\" (UID: \"57e39a4b-a64e-4689-8b46-15c4a3e2ce48\") " Oct 30 13:04:58.507576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.507551 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.507649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.507582 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44trj\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-kube-api-access-44trj\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.507649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.507599 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25132bf1-77b7-4acb-940e-d36e07a12a76-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.507649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.507612 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25132bf1-77b7-4acb-940e-d36e07a12a76-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.507649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.507627 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe03a9a4-eae9-418e-b91c-73a175fc4804-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.507649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.507640 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frrxc\" (UniqueName: \"kubernetes.io/projected/25132bf1-77b7-4acb-940e-d36e07a12a76-kube-api-access-frrxc\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.508585 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.508235 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-socket" pod "52c06c8e-6813-483b-9ea1-de079980854a" (UID: "52c06c8e-6813-483b-9ea1-de079980854a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.509592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.509390 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-certs" pod "57e39a4b-a64e-4689-8b46-15c4a3e2ce48" (UID: "57e39a4b-a64e-4689-8b46-15c4a3e2ce48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.509704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.509639 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-certs" pod "52c06c8e-6813-483b-9ea1-de079980854a" (UID: "52c06c8e-6813-483b-9ea1-de079980854a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.510075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.509803 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-data" pod "57e39a4b-a64e-4689-8b46-15c4a3e2ce48" (UID: "57e39a4b-a64e-4689-8b46-15c4a3e2ce48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.510319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.510217 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c06c8e-6813-483b-9ea1-de079980854a-istiod-ca-cert" pod "52c06c8e-6813-483b-9ea1-de079980854a" (UID: "52c06c8e-6813-483b-9ea1-de079980854a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:58.510496 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.510469 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-credential-socket" pod "57e39a4b-a64e-4689-8b46-15c4a3e2ce48" (UID: "57e39a4b-a64e-4689-8b46-15c4a3e2ce48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.510587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.510518 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-data" pod "52c06c8e-6813-483b-9ea1-de079980854a" (UID: "52c06c8e-6813-483b-9ea1-de079980854a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.510587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.510555 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-socket" pod "57e39a4b-a64e-4689-8b46-15c4a3e2ce48" (UID: "57e39a4b-a64e-4689-8b46-15c4a3e2ce48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.511028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.510968 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istiod-ca-cert" pod "57e39a4b-a64e-4689-8b46-15c4a3e2ce48" (UID: "57e39a4b-a64e-4689-8b46-15c4a3e2ce48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:58.511217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.511166 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-credential-socket" pod "52c06c8e-6813-483b-9ea1-de079980854a" (UID: "52c06c8e-6813-483b-9ea1-de079980854a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.511217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.511188 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-kube-api-access-g59mq" pod "52c06c8e-6813-483b-9ea1-de079980854a" (UID: "52c06c8e-6813-483b-9ea1-de079980854a"). InnerVolumeSpecName "kube-api-access-g59mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.513711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.513568 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-envoy" pod "52c06c8e-6813-483b-9ea1-de079980854a" (UID: "52c06c8e-6813-483b-9ea1-de079980854a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.513711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.513673 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52c06c8e-6813-483b-9ea1-de079980854a-istio-podinfo" pod "52c06c8e-6813-483b-9ea1-de079980854a" (UID: "52c06c8e-6813-483b-9ea1-de079980854a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:58.514196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.514172 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-kube-api-access-lk8mt" pod "57e39a4b-a64e-4689-8b46-15c4a3e2ce48" (UID: "57e39a4b-a64e-4689-8b46-15c4a3e2ce48"). InnerVolumeSpecName "kube-api-access-lk8mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.514535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.514359 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-token" pod "57e39a4b-a64e-4689-8b46-15c4a3e2ce48" (UID: "57e39a4b-a64e-4689-8b46-15c4a3e2ce48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.514664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.514645 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-podinfo" pod "57e39a4b-a64e-4689-8b46-15c4a3e2ce48" (UID: "57e39a4b-a64e-4689-8b46-15c4a3e2ce48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:58.515473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.515450 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-istio-token" pod "52c06c8e-6813-483b-9ea1-de079980854a" (UID: "52c06c8e-6813-483b-9ea1-de079980854a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:58.516041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.516022 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-envoy" pod "57e39a4b-a64e-4689-8b46-15c4a3e2ce48" (UID: "57e39a4b-a64e-4689-8b46-15c4a3e2ce48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:58.536425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.536390 5515 generic.go:356] "Generic (PLEG): container finished" podID="52c06c8e-6813-483b-9ea1-de079980854a" containerID="c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a" exitCode=0 Oct 30 13:04:58.536604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.536450 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" Oct 30 13:04:58.536604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.536453 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" event={"ID":"52c06c8e-6813-483b-9ea1-de079980854a","Type":"ContainerDied","Data":"c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a"} Oct 30 13:04:58.536604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.536559 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4" event={"ID":"52c06c8e-6813-483b-9ea1-de079980854a","Type":"ContainerDied","Data":"6ab53fe6d40d0672ce9c36e9ae1f6632d6e47972d3dc83e57516f90b992a8cb2"} Oct 30 13:04:58.536604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.536594 5515 scope.go:122] "RemoveContainer" containerID="c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a" Oct 30 13:04:58.538408 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.538379 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" event={"ID":"d8e90e60-fab3-49e7-9541-6c03658a7114","Type":"ContainerStarted","Data":"f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8"} Oct 30 13:04:58.538529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.538446 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" event={"ID":"d8e90e60-fab3-49e7-9541-6c03658a7114","Type":"ContainerStarted","Data":"363cd9db373884fe674ac46e3ad4fff09456dafa17a28f53225ffe905cce18bc"} Oct 30 13:04:58.538529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.538479 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:04:58.539870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.539848 5515 generic.go:356] "Generic (PLEG): container finished" podID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerID="34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22" exitCode=0 Oct 30 13:04:58.539984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.539918 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" Oct 30 13:04:58.539984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.539939 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" event={"ID":"25132bf1-77b7-4acb-940e-d36e07a12a76","Type":"ContainerDied","Data":"34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22"} Oct 30 13:04:58.539984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.539980 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz" event={"ID":"25132bf1-77b7-4acb-940e-d36e07a12a76","Type":"ContainerDied","Data":"9bd2408e8eb4a7e2d89ea23a388d9e3cbc5c632044f98564d26ce4f86bd50470"} Oct 30 13:04:58.541892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.541866 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" event={"ID":"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da","Type":"ContainerStarted","Data":"b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9"} Oct 30 13:04:58.542013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.541945 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:04:58.544547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.544521 5515 generic.go:356] "Generic (PLEG): container finished" podID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerID="d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5" exitCode=0 Oct 30 13:04:58.544655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.544563 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" Oct 30 13:04:58.544655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.544598 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" event={"ID":"fe03a9a4-eae9-418e-b91c-73a175fc4804","Type":"ContainerDied","Data":"d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5"} Oct 30 13:04:58.544655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.544640 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd" event={"ID":"fe03a9a4-eae9-418e-b91c-73a175fc4804","Type":"ContainerDied","Data":"f5862a328f0a1a169f1f05cd71ef858048f6a01f8c2870a7ed13a3f953b9672b"} Oct 30 13:04:58.547832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.547739 5515 generic.go:356] "Generic (PLEG): container finished" podID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerID="cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf" exitCode=0 Oct 30 13:04:58.547832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.547776 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" Oct 30 13:04:58.547832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.547784 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" event={"ID":"57e39a4b-a64e-4689-8b46-15c4a3e2ce48","Type":"ContainerDied","Data":"cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf"} Oct 30 13:04:58.547832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.547822 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d" event={"ID":"57e39a4b-a64e-4689-8b46-15c4a3e2ce48","Type":"ContainerDied","Data":"b1e1b04fd73b51f773527b80d77c9397ed6bf6bf16450b70a0cc054f6b5395e2"} Oct 30 13:04:58.549405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.549386 5515 scope.go:122] "RemoveContainer" containerID="c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a" Oct 30 13:04:58.549695 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:58.549675 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a\": container with ID starting with c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a not found: ID does not exist" containerID="c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a" Oct 30 13:04:58.549793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.549707 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a"} err="failed to get container status \"c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a\": rpc error: code = NotFound desc = could not find container \"c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a\": container with ID starting with c71f2a4c275ff6a54c44196605daada8fd8e5526670d501e7bb5b579d3ec938a not found: ID does not exist" Oct 30 13:04:58.549793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.549731 5515 scope.go:122] "RemoveContainer" containerID="34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22" Oct 30 13:04:58.558012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.557985 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:58.561663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.561641 5515 scope.go:122] "RemoveContainer" containerID="34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22" Oct 30 13:04:58.561929 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:58.561909 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22\": container with ID starting with 34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22 not found: ID does not exist" containerID="34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22" Oct 30 13:04:58.562000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.561936 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22"} err="failed to get container status \"34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22\": rpc error: code = NotFound desc = could not find container \"34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22\": container with ID starting with 34f820fa59aff7bb4afc9d66c33e2e1101f76b4d12d83d142caf114bee7d9e22 not found: ID does not exist" Oct 30 13:04:58.562000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.561952 5515 scope.go:122] "RemoveContainer" containerID="d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5" Oct 30 13:04:58.572087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.572066 5515 scope.go:122] "RemoveContainer" containerID="d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5" Oct 30 13:04:58.572461 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:58.572432 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5\": container with ID starting with d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5 not found: ID does not exist" containerID="d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5" Oct 30 13:04:58.572542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.572464 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5"} err="failed to get container status \"d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5\": rpc error: code = NotFound desc = could not find container \"d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5\": container with ID starting with d331cffcab644150db0023a93ed2c472afec2ff01d9be4609686984e4bcc7ed5 not found: ID does not exist" Oct 30 13:04:58.572542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.572484 5515 scope.go:122] "RemoveContainer" containerID="cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf" Oct 30 13:04:58.586603 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.586575 5515 scope.go:122] "RemoveContainer" containerID="cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf" Oct 30 13:04:58.587057 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:58.587026 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf\": container with ID starting with cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf not found: ID does not exist" containerID="cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf" Oct 30 13:04:58.587188 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.587067 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf"} err="failed to get container status \"cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf\": rpc error: code = NotFound desc = could not find container \"cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf\": container with ID starting with cea0aaebd70642659363e4007f18de4fb61327a0dce56e5ed31aec5ae99b0ddf not found: ID does not exist" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611121 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611160 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c06c8e-6813-483b-9ea1-de079980854a-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611177 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611190 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611202 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611217 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk8mt\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-kube-api-access-lk8mt\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611231 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611245 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611258 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611289 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611301 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611313 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611326 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c06c8e-6813-483b-9ea1-de079980854a-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611339 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611353 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c06c8e-6813-483b-9ea1-de079980854a-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611366 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611379 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e39a4b-a64e-4689-8b46-15c4a3e2ce48-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.637488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.611394 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g59mq\" (UniqueName: \"kubernetes.io/projected/52c06c8e-6813-483b-9ea1-de079980854a-kube-api-access-g59mq\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:04:58.660429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.660359 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4"] Oct 30 13:04:58.666213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.666157 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-kfhm4"] Oct 30 13:04:58.687556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.687505 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" podStartSLOduration=2.687489207 podStartE2EDuration="2.687489207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:58.68469471 +0000 UTC m=+1916.811495608" watchObservedRunningTime="2026-10-30 13:04:58.687489207 +0000 UTC m=+1916.814290104" Oct 30 13:04:58.764428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.764323 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" podStartSLOduration=2.764308009 podStartE2EDuration="2.764308009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:58.761845157 +0000 UTC m=+1916.888646055" watchObservedRunningTime="2026-10-30 13:04:58.764308009 +0000 UTC m=+1916.891108907" Oct 30 13:04:58.815283 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.815240 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d"] Oct 30 13:04:58.819869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.819839 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-6gx5d"] Oct 30 13:04:58.858761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.858730 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd"] Oct 30 13:04:58.861824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.861797 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sgkrd"] Oct 30 13:04:58.927809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.926717 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz"] Oct 30 13:04:58.940307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:58.937807 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-sslhz"] Oct 30 13:04:59.018171 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:04:59.018095 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:04:59.626489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.626422 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:59.634424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.634389 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:04:59.683889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.683858 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7"] Oct 30 13:04:59.684541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684527 5515 cpu_manager.go:414] "containerMap: removing container" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" Oct 30 13:04:59.684583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684543 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" Oct 30 13:04:59.684583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684568 5515 cpu_manager.go:414] "containerMap: removing container" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" Oct 30 13:04:59.684583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684573 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" Oct 30 13:04:59.684696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684588 5515 cpu_manager.go:414] "containerMap: removing container" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" Oct 30 13:04:59.684696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684593 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" Oct 30 13:04:59.684696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684601 5515 cpu_manager.go:414] "containerMap: removing container" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" Oct 30 13:04:59.684696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684606 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" Oct 30 13:04:59.684696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684685 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" containerName="istio-proxy" Oct 30 13:04:59.684853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684699 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" containerName="istio-proxy" Oct 30 13:04:59.684853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684709 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" containerName="istio-proxy" Oct 30 13:04:59.684853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.684715 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c06c8e-6813-483b-9ea1-de079980854a" containerName="istio-proxy" Oct 30 13:04:59.689577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.689547 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:04:59.709210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.709177 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7"] Oct 30 13:04:59.711867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.711832 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f9c553-dockercfg-99r2p\"" Oct 30 13:04:59.715678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.715645 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f9c553\"" Oct 30 13:04:59.744483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.744437 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n"] Oct 30 13:04:59.749671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.749640 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:04:59.788278 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.788192 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n"] Oct 30 13:04:59.956646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.956613 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:59.958230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.958188 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8ls\" (UniqueName: \"kubernetes.io/projected/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-kube-api-access-bv8ls\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7\" (UID: \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:04:59.958700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.958676 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d2hr\" (UniqueName: \"kubernetes.io/projected/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-kube-api-access-4d2hr\") pod \"payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n\" (UID: \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:04:59.958936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.958903 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7\" (UID: \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:04:59.959232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:04:59.959215 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n\" (UID: \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:00.044003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.043966 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25132bf1-77b7-4acb-940e-d36e07a12a76" path="/var/lib/kubelet/pods/25132bf1-77b7-4acb-940e-d36e07a12a76/volumes" Oct 30 13:05:00.044657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.044633 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c06c8e-6813-483b-9ea1-de079980854a" path="/var/lib/kubelet/pods/52c06c8e-6813-483b-9ea1-de079980854a/volumes" Oct 30 13:05:00.045104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.045089 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e39a4b-a64e-4689-8b46-15c4a3e2ce48" path="/var/lib/kubelet/pods/57e39a4b-a64e-4689-8b46-15c4a3e2ce48/volumes" Oct 30 13:05:00.046058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.046037 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe03a9a4-eae9-418e-b91c-73a175fc4804" path="/var/lib/kubelet/pods/fe03a9a4-eae9-418e-b91c-73a175fc4804/volumes" Oct 30 13:05:00.060967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.060925 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8ls\" (UniqueName: \"kubernetes.io/projected/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-kube-api-access-bv8ls\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7\" (UID: \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:05:00.061131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.060972 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d2hr\" (UniqueName: \"kubernetes.io/projected/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-kube-api-access-4d2hr\") pod \"payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n\" (UID: \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:00.061131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.061011 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7\" (UID: \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:05:00.061131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.061078 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n\" (UID: \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:00.062014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.061989 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7\" (UID: \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:05:00.062064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.061989 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n\" (UID: \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:00.084892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.084864 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8ls\" (UniqueName: \"kubernetes.io/projected/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-kube-api-access-bv8ls\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7\" (UID: \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:05:00.085027 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.084965 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d2hr\" (UniqueName: \"kubernetes.io/projected/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-kube-api-access-4d2hr\") pod \"payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n\" (UID: \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:00.258094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.257995 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:00.304575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.304542 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:05:00.427615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.425124 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:00.676695 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:05:00.676665 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267e14eb_68e6_49a3_bbf4_a500f9a6e66e.slice/crio-d443d363864aa6acc91a19233a4bb82c5511fb9ba5b578c210241809fe5def16 WatchSource:0}: Error finding container d443d363864aa6acc91a19233a4bb82c5511fb9ba5b578c210241809fe5def16: Status 404 returned error can't find the container with id d443d363864aa6acc91a19233a4bb82c5511fb9ba5b578c210241809fe5def16 Oct 30 13:05:00.677052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.676937 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n"] Oct 30 13:05:00.720378 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:05:00.720349 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd55a2e2_aaaf_4a2f_b99d_5254c25c77f7.slice/crio-5a449f160244a2c69cad1dee15c8620acd3a3c5efbd5f7fa4766d5ed174de95e WatchSource:0}: Error finding container 5a449f160244a2c69cad1dee15c8620acd3a3c5efbd5f7fa4766d5ed174de95e: Status 404 returned error can't find the container with id 5a449f160244a2c69cad1dee15c8620acd3a3c5efbd5f7fa4766d5ed174de95e Oct 30 13:05:00.720949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:00.720874 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7"] Oct 30 13:05:01.578291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:01.578089 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" event={"ID":"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7","Type":"ContainerStarted","Data":"db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34"} Oct 30 13:05:01.578291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:01.578153 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" event={"ID":"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7","Type":"ContainerStarted","Data":"5a449f160244a2c69cad1dee15c8620acd3a3c5efbd5f7fa4766d5ed174de95e"} Oct 30 13:05:01.578291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:01.578225 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:05:01.579953 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:01.579921 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" event={"ID":"267e14eb-68e6-49a3-bbf4-a500f9a6e66e","Type":"ContainerStarted","Data":"015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5"} Oct 30 13:05:01.580357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:01.579978 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" event={"ID":"267e14eb-68e6-49a3-bbf4-a500f9a6e66e","Type":"ContainerStarted","Data":"d443d363864aa6acc91a19233a4bb82c5511fb9ba5b578c210241809fe5def16"} Oct 30 13:05:01.580357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:01.580127 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:01.633007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:01.632937 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" podStartSLOduration=2.632916116 podStartE2EDuration="2.632916116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:01.631690805 +0000 UTC m=+1919.758491717" watchObservedRunningTime="2026-10-30 13:05:01.632916116 +0000 UTC m=+1919.759717015" Oct 30 13:05:01.684694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:01.684619 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" podStartSLOduration=2.68459937 podStartE2EDuration="2.68459937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:01.681426475 +0000 UTC m=+1919.808227375" watchObservedRunningTime="2026-10-30 13:05:01.68459937 +0000 UTC m=+1919.811400270" Oct 30 13:05:02.164293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:02.150532 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" podUID="9d9a314f-a99e-4e31-8adf-aebc11adf7fc" containerName="payload-processing" containerID="cri-o://812660ba763aa3124213f3360b118088131cc06e432101ed28e38b03899349e6" gracePeriod=25 Oct 30 13:05:02.786442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:02.786347 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:02.789307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:02.788708 5515 generic.go:356] "Generic (PLEG): container finished" podID="9d9a314f-a99e-4e31-8adf-aebc11adf7fc" containerID="812660ba763aa3124213f3360b118088131cc06e432101ed28e38b03899349e6" exitCode=0 Oct 30 13:05:02.790236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:02.789937 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" event={"ID":"9d9a314f-a99e-4e31-8adf-aebc11adf7fc","Type":"ContainerDied","Data":"812660ba763aa3124213f3360b118088131cc06e432101ed28e38b03899349e6"} Oct 30 13:05:02.991645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:02.991617 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:05:03.065298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.062374 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-kube-api-access-8kjsb\" (UniqueName: \"kubernetes.io/projected/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-kube-api-access-8kjsb\") pod \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\" (UID: \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\") " Oct 30 13:05:03.065298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.062997 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-plugins-config-volume\") pod \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\" (UID: \"9d9a314f-a99e-4e31-8adf-aebc11adf7fc\") " Oct 30 13:05:03.065298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.064244 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-plugins-config-volume" pod "9d9a314f-a99e-4e31-8adf-aebc11adf7fc" (UID: "9d9a314f-a99e-4e31-8adf-aebc11adf7fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:03.074829 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.074793 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-kube-api-access-8kjsb" pod "9d9a314f-a99e-4e31-8adf-aebc11adf7fc" (UID: "9d9a314f-a99e-4e31-8adf-aebc11adf7fc"). InnerVolumeSpecName "kube-api-access-8kjsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:03.193061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.193016 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjsb\" (UniqueName: \"kubernetes.io/projected/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-kube-api-access-8kjsb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:03.193235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.193083 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9a314f-a99e-4e31-8adf-aebc11adf7fc-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:03.798747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.798687 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" event={"ID":"9d9a314f-a99e-4e31-8adf-aebc11adf7fc","Type":"ContainerDied","Data":"e6000a6c9c91070362c5f699da867bf65c1088c010c5046dff8715121ca91f23"} Oct 30 13:05:03.799317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.798810 5515 scope.go:122] "RemoveContainer" containerID="812660ba763aa3124213f3360b118088131cc06e432101ed28e38b03899349e6" Oct 30 13:05:03.799317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.798998 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt" Oct 30 13:05:03.885634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.885601 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt"] Oct 30 13:05:03.898795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:03.898762 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-fd7766b56-fh9xt"] Oct 30 13:05:04.003450 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:04.003418 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9a314f-a99e-4e31-8adf-aebc11adf7fc" path="/var/lib/kubelet/pods/9d9a314f-a99e-4e31-8adf-aebc11adf7fc/volumes" Oct 30 13:05:04.795541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:04.790132 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:05.059332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.059210 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:05.384551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.384520 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22"] Oct 30 13:05:05.385393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.385372 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9d9a314f-a99e-4e31-8adf-aebc11adf7fc" containerName="payload-processing" Oct 30 13:05:05.385393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.385393 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9a314f-a99e-4e31-8adf-aebc11adf7fc" containerName="payload-processing" Oct 30 13:05:05.385548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.385537 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9a314f-a99e-4e31-8adf-aebc11adf7fc" containerName="payload-processing" Oct 30 13:05:05.389233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.389211 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:05.412609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.412576 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22"] Oct 30 13:05:05.420026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.419982 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-405eb1-dockercfg-d584m\"" Oct 30 13:05:05.420204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.420185 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-405eb1\"" Oct 30 13:05:05.568225 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.568188 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458rw\" (UniqueName: \"kubernetes.io/projected/60554da6-b176-4d64-a077-0dc191618029-kube-api-access-458rw\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22\" (UID: \"60554da6-b176-4d64-a077-0dc191618029\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:05.568545 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.568522 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60554da6-b176-4d64-a077-0dc191618029-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22\" (UID: \"60554da6-b176-4d64-a077-0dc191618029\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:05.669498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.669399 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60554da6-b176-4d64-a077-0dc191618029-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22\" (UID: \"60554da6-b176-4d64-a077-0dc191618029\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:05.669657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.669505 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458rw\" (UniqueName: \"kubernetes.io/projected/60554da6-b176-4d64-a077-0dc191618029-kube-api-access-458rw\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22\" (UID: \"60554da6-b176-4d64-a077-0dc191618029\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:05.670490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.670457 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60554da6-b176-4d64-a077-0dc191618029-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22\" (UID: \"60554da6-b176-4d64-a077-0dc191618029\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:05.693543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.693509 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458rw\" (UniqueName: \"kubernetes.io/projected/60554da6-b176-4d64-a077-0dc191618029-kube-api-access-458rw\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22\" (UID: \"60554da6-b176-4d64-a077-0dc191618029\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:05.703507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.703478 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:05.768195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.768159 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm"] Oct 30 13:05:05.773980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.773955 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:05.803253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.803222 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm"] Oct 30 13:05:05.873491 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.873456 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrnm\" (UniqueName: \"kubernetes.io/projected/25749efb-4232-406d-aae6-6e6a802b2333-kube-api-access-lwrnm\") pod \"payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm\" (UID: \"25749efb-4232-406d-aae6-6e6a802b2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:05.873669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.873648 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25749efb-4232-406d-aae6-6e6a802b2333-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm\" (UID: \"25749efb-4232-406d-aae6-6e6a802b2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:05.974911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.974835 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrnm\" (UniqueName: \"kubernetes.io/projected/25749efb-4232-406d-aae6-6e6a802b2333-kube-api-access-lwrnm\") pod \"payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm\" (UID: \"25749efb-4232-406d-aae6-6e6a802b2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:05.975053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.974959 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25749efb-4232-406d-aae6-6e6a802b2333-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm\" (UID: \"25749efb-4232-406d-aae6-6e6a802b2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:05.975682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.975662 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25749efb-4232-406d-aae6-6e6a802b2333-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm\" (UID: \"25749efb-4232-406d-aae6-6e6a802b2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:05.999411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:05.999378 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrnm\" (UniqueName: \"kubernetes.io/projected/25749efb-4232-406d-aae6-6e6a802b2333-kube-api-access-lwrnm\") pod \"payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm\" (UID: \"25749efb-4232-406d-aae6-6e6a802b2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:06.013456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:06.013425 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22"] Oct 30 13:05:06.014084 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:05:06.014058 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60554da6_b176_4d64_a077_0dc191618029.slice/crio-f3c08bc88808f40ab13088c736d7626225c3f355e943af95de46ae0b94f260db WatchSource:0}: Error finding container f3c08bc88808f40ab13088c736d7626225c3f355e943af95de46ae0b94f260db: Status 404 returned error can't find the container with id f3c08bc88808f40ab13088c736d7626225c3f355e943af95de46ae0b94f260db Oct 30 13:05:06.016241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:06.016219 5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:05:06.105046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:06.104649 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:06.163510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:06.163472 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:06.479256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:06.479222 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm"] Oct 30 13:05:06.479425 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:05:06.479320 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25749efb_4232_406d_aae6_6e6a802b2333.slice/crio-ec60b63901a35c81d3c06597d269fd5cfcaef7d47209da3d3a288877817121a0 WatchSource:0}: Error finding container ec60b63901a35c81d3c06597d269fd5cfcaef7d47209da3d3a288877817121a0: Status 404 returned error can't find the container with id ec60b63901a35c81d3c06597d269fd5cfcaef7d47209da3d3a288877817121a0 Oct 30 13:05:06.834821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:06.834734 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" event={"ID":"25749efb-4232-406d-aae6-6e6a802b2333","Type":"ContainerStarted","Data":"ec60b63901a35c81d3c06597d269fd5cfcaef7d47209da3d3a288877817121a0"} Oct 30 13:05:06.836670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:06.836635 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" event={"ID":"60554da6-b176-4d64-a077-0dc191618029","Type":"ContainerStarted","Data":"f3c08bc88808f40ab13088c736d7626225c3f355e943af95de46ae0b94f260db"} Oct 30 13:05:07.850879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:07.850837 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" event={"ID":"25749efb-4232-406d-aae6-6e6a802b2333","Type":"ContainerStarted","Data":"f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c"} Oct 30 13:05:07.851371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:07.851134 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:07.852691 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:07.852665 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" event={"ID":"60554da6-b176-4d64-a077-0dc191618029","Type":"ContainerStarted","Data":"dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3"} Oct 30 13:05:07.852824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:07.852754 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:07.908590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:07.908534 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" podStartSLOduration=1.654614099 podStartE2EDuration="2.908518796s" totalImagesPullingTime="1.253904697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:07.90723405 +0000 UTC m=+1926.034034952" watchObservedRunningTime="2026-10-30 13:05:07.908518796 +0000 UTC m=+1926.035319694" Oct 30 13:05:07.957743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:07.957627 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" podStartSLOduration=1.299201597 podStartE2EDuration="2.957612653s" totalImagesPullingTime="1.658411056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:07.955211173 +0000 UTC m=+1926.082012070" watchObservedRunningTime="2026-10-30 13:05:07.957612653 +0000 UTC m=+1926.084413590" Oct 30 13:05:07.999447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:07.999413 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:09.580292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:09.576656 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:05:09.580292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:09.579426 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:05:09.612519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:09.611093 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:10.158230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:10.158197 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:11.218290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:11.215003 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:12.793790 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:12.793762 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:05:12.794195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:12.793867 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:13.269881 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:13.267750 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:15.269279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.269245 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg"] Oct 30 13:05:15.274019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.273993 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.309062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.309022 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg"] Oct 30 13:05:15.309244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.309184 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a121dd-serving-cert\"" Oct 30 13:05:15.340674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.340640 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-metrics-certs\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.340870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.340728 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994b01e-7756-40c1-87df-3e026a6043f6-openshift-service-ca\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.340870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.340755 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7z7h\" (UniqueName: \"kubernetes.io/projected/b994b01e-7756-40c1-87df-3e026a6043f6-kube-api-access-h7z7h\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.340870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.340817 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.442110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.442066 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-metrics-certs\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.442110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.442119 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994b01e-7756-40c1-87df-3e026a6043f6-openshift-service-ca\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.442507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.442153 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7z7h\" (UniqueName: \"kubernetes.io/projected/b994b01e-7756-40c1-87df-3e026a6043f6-kube-api-access-h7z7h\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.442507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.442257 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.442507 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:15.442496 5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-serving-cert: secret "maas-api-e2e-api-b-a121dd-serving-cert" not found Oct 30 13:05:15.442667 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:15.442573 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls podName:b994b01e-7756-40c1-87df-3e026a6043f6 nodeName:}" failed. No retries permitted until 2026-10-30 13:05:15.942551906 +0000 UTC m=+1934.069352783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls") pod "maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" (UID: "b994b01e-7756-40c1-87df-3e026a6043f6") : secret "maas-api-e2e-api-b-a121dd-serving-cert" not found Oct 30 13:05:15.442972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.442954 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994b01e-7756-40c1-87df-3e026a6043f6-openshift-service-ca\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.444803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.444782 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-metrics-certs\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.491032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.490913 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:15.510291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.507019 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7z7h\" (UniqueName: \"kubernetes.io/projected/b994b01e-7756-40c1-87df-3e026a6043f6-kube-api-access-h7z7h\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.739988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.739951 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:15.949366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.949321 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.951845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.951823 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls\") pod \"maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:15.994148 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:15.994058 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:16.188757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:16.188715 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:16.675848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:16.675815 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg"] Oct 30 13:05:16.926514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:16.926415 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" event={"ID":"b994b01e-7756-40c1-87df-3e026a6043f6","Type":"ContainerStarted","Data":"b6dbffc01e75a463727e80f49494ea15df5df739fe2113771c5be103ab4761bb"} Oct 30 13:05:17.615570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:17.615538 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:17.934562 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:17.934520 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" event={"ID":"b994b01e-7756-40c1-87df-3e026a6043f6","Type":"ContainerStarted","Data":"5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1"} Oct 30 13:05:17.935050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:17.934596 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:17.979024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:17.978973 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" podStartSLOduration=2.6227456350000002 podStartE2EDuration="2.978958849s" totalImagesPullingTime="356.213214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:17.978475762 +0000 UTC m=+1936.105276663" watchObservedRunningTime="2026-10-30 13:05:17.978958849 +0000 UTC m=+1936.105759746" Oct 30 13:05:18.691389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:18.691347 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:18.882481 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:18.882444 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:05:18.884585 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:18.884556 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:05:18.928864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:18.928832 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:19.576807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.576777 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt"] Oct 30 13:05:19.581180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.581160 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.605166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.605128 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt"] Oct 30 13:05:19.609436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.609406 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fa855c-serving-cert\"" Oct 30 13:05:19.640074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.640036 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367c5bd-d5bf-47be-881f-21427af0aad2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.640237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.640100 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.640311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.640295 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbg6x\" (UniqueName: \"kubernetes.io/projected/c367c5bd-d5bf-47be-881f-21427af0aad2-kube-api-access-xbg6x\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.640406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.640393 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-metrics-certs\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.741382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.741337 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-metrics-certs\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.741595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.741415 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367c5bd-d5bf-47be-881f-21427af0aad2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.741595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.741458 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.741595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.741559 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbg6x\" (UniqueName: \"kubernetes.io/projected/c367c5bd-d5bf-47be-881f-21427af0aad2-kube-api-access-xbg6x\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.742151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.742126 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367c5bd-d5bf-47be-881f-21427af0aad2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.743848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.743819 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-metrics-certs\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.743938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.743916 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.766990 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.766961 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbg6x\" (UniqueName: \"kubernetes.io/projected/c367c5bd-d5bf-47be-881f-21427af0aad2-kube-api-access-xbg6x\") pod \"maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:19.860779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.859946 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:19.898197 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:19.898158 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:20.105889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:20.105855 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:20.218972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:20.218943 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt"] Oct 30 13:05:20.219240 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:05:20.219213 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc367c5bd_d5bf_47be_881f_21427af0aad2.slice/crio-f3c8500d23019f16cfda9aa2f4644594b5ce815f4bff1c562695a04c0f14ba6b WatchSource:0}: Error finding container f3c8500d23019f16cfda9aa2f4644594b5ce815f4bff1c562695a04c0f14ba6b: Status 404 returned error can't find the container with id f3c8500d23019f16cfda9aa2f4644594b5ce815f4bff1c562695a04c0f14ba6b Oct 30 13:05:21.006977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:21.006939 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" event={"ID":"c367c5bd-d5bf-47be-881f-21427af0aad2","Type":"ContainerStarted","Data":"cd002dfa489a89385e47361b4d66a088c02615bde363956d7c16ee64538c6576"} Oct 30 13:05:21.007443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:21.007124 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:21.007443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:21.007171 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" event={"ID":"c367c5bd-d5bf-47be-881f-21427af0aad2","Type":"ContainerStarted","Data":"f3c8500d23019f16cfda9aa2f4644594b5ce815f4bff1c562695a04c0f14ba6b"} Oct 30 13:05:21.055029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:21.054967 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" podStartSLOduration=1.6646118159999999 podStartE2EDuration="2.054950011s" totalImagesPullingTime="390.338195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:21.05255295 +0000 UTC m=+1939.179353862" watchObservedRunningTime="2026-10-30 13:05:21.054950011 +0000 UTC m=+1939.181750909" Oct 30 13:05:21.716718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:21.716678 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:23.173995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.173937 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg"] Oct 30 13:05:23.178855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.178828 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.199057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.199023 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg"] Oct 30 13:05:23.249934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.249903 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f9c553-serving-cert\"" Oct 30 13:05:23.348668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.348630 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.348861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.348709 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.348861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.348740 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fz22\" (UniqueName: \"kubernetes.io/projected/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-kube-api-access-6fz22\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.348979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.348918 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.386872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.386839 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:23.454481 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.454376 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.454807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.454777 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.454975 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.454955 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fz22\" (UniqueName: \"kubernetes.io/projected/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-kube-api-access-6fz22\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.472619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.472576 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.473651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.464312 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.473999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.456239 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.484131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.484083 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.508999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.508927 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fz22\" (UniqueName: \"kubernetes.io/projected/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-kube-api-access-6fz22\") pod \"maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.655521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.655475 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:23.795202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.795122 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:23.984480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:23.984449 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:05:24.106643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:24.106615 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg"] Oct 30 13:05:24.107086 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:05:24.107063 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6dfb2a_cf2e_4bc0_8df3_2d92307afa9e.slice/crio-9fc5a8afcde255ef041d1218f3c3965061656d50e6b223f06367aaec05f0f128 WatchSource:0}: Error finding container 9fc5a8afcde255ef041d1218f3c3965061656d50e6b223f06367aaec05f0f128: Status 404 returned error can't find the container with id 9fc5a8afcde255ef041d1218f3c3965061656d50e6b223f06367aaec05f0f128 Oct 30 13:05:25.037372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:25.037324 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" event={"ID":"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e","Type":"ContainerStarted","Data":"6797a2b901ce0bdec1657c31068e5bd337b8b514e18ee6a0bcde139cfda8a050"} Oct 30 13:05:25.037817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:25.037424 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" event={"ID":"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e","Type":"ContainerStarted","Data":"9fc5a8afcde255ef041d1218f3c3965061656d50e6b223f06367aaec05f0f128"} Oct 30 13:05:25.037817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:25.037563 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:25.084105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:25.084035 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" podStartSLOduration=1.6759232210000001 podStartE2EDuration="2.084020062s" totalImagesPullingTime="408.096841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:25.081230982 +0000 UTC m=+1943.208031880" watchObservedRunningTime="2026-10-30 13:05:25.084020062 +0000 UTC m=+1943.210820962" Oct 30 13:05:25.235297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:25.231525 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:26.827707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:26.827656 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:27.019646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:27.019618 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:05:28.412977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:28.412877 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:28.990553 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:28.990517 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:05:30.001712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:30.001674 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:31.060757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:31.060718 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:05:32.184771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:32.184735 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:32.877985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:32.877946 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:33.106082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:33.106047 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:33.842007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:33.841974 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:35.733648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:35.733616 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:37.293812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:37.293778 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:39.067104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.067062 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz"] Oct 30 13:05:39.067656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.067329 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" podUID="61452246-0bb8-4c65-bdbd-b2c3f53977cb" containerName="maas-api" containerID="cri-o://a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05" gracePeriod=30 Oct 30 13:05:39.262330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.262216 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:39.366469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.366442 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:05:39.486281 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.486226 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls\") pod \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " Oct 30 13:05:39.486507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.486301 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61452246-0bb8-4c65-bdbd-b2c3f53977cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61452246-0bb8-4c65-bdbd-b2c3f53977cb-openshift-service-ca\") pod \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " Oct 30 13:05:39.486507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.486370 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61452246-0bb8-4c65-bdbd-b2c3f53977cb-kube-api-access-xdw85\" (UniqueName: \"kubernetes.io/projected/61452246-0bb8-4c65-bdbd-b2c3f53977cb-kube-api-access-xdw85\") pod \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " Oct 30 13:05:39.486507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.486444 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-metrics-certs\") pod \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\" (UID: \"61452246-0bb8-4c65-bdbd-b2c3f53977cb\") " Oct 30 13:05:39.487037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.486756 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61452246-0bb8-4c65-bdbd-b2c3f53977cb-openshift-service-ca" pod "61452246-0bb8-4c65-bdbd-b2c3f53977cb" (UID: "61452246-0bb8-4c65-bdbd-b2c3f53977cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:39.487167 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.487141 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61452246-0bb8-4c65-bdbd-b2c3f53977cb-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:39.489308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.489229 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls" pod "61452246-0bb8-4c65-bdbd-b2c3f53977cb" (UID: "61452246-0bb8-4c65-bdbd-b2c3f53977cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:39.489308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.489248 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-metrics-certs" pod "61452246-0bb8-4c65-bdbd-b2c3f53977cb" (UID: "61452246-0bb8-4c65-bdbd-b2c3f53977cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:39.493432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.493406 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61452246-0bb8-4c65-bdbd-b2c3f53977cb-kube-api-access-xdw85" pod "61452246-0bb8-4c65-bdbd-b2c3f53977cb" (UID: "61452246-0bb8-4c65-bdbd-b2c3f53977cb"). InnerVolumeSpecName "kube-api-access-xdw85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:39.521778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.521743 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:39.599375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.588485 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:39.599375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.588532 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdw85\" (UniqueName: \"kubernetes.io/projected/61452246-0bb8-4c65-bdbd-b2c3f53977cb-kube-api-access-xdw85\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:39.599375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.588553 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61452246-0bb8-4c65-bdbd-b2c3f53977cb-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:39.773993 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:39.773957 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:40.041338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.038190 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:40.184877 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.184843 5515 generic.go:356] "Generic (PLEG): container finished" podID="61452246-0bb8-4c65-bdbd-b2c3f53977cb" containerID="a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05" exitCode=0 Oct 30 13:05:40.185346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.184895 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" Oct 30 13:05:40.185346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.184929 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" event={"ID":"61452246-0bb8-4c65-bdbd-b2c3f53977cb","Type":"ContainerDied","Data":"a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05"} Oct 30 13:05:40.185346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.184984 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz" event={"ID":"61452246-0bb8-4c65-bdbd-b2c3f53977cb","Type":"ContainerDied","Data":"f99c1c41a825cea72a9d9b50c50ca591a843545c64ff524252a0124ba5426617"} Oct 30 13:05:40.185346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.185007 5515 scope.go:122] "RemoveContainer" containerID="a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05" Oct 30 13:05:40.197942 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.197922 5515 scope.go:122] "RemoveContainer" containerID="a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05" Oct 30 13:05:40.198256 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:40.198236 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05\": container with ID starting with a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05 not found: ID does not exist" containerID="a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05" Oct 30 13:05:40.198354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.198299 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05"} err="failed to get container status \"a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05\": rpc error: code = NotFound desc = could not find container \"a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05\": container with ID starting with a78980a2166752891fe387a0fea6104da227db2fef4b31479b03bed47bed3d05 not found: ID does not exist" Oct 30 13:05:40.239867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.239833 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz"] Oct 30 13:05:40.246571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.246543 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9368ce-64b977d58c-l22xz"] Oct 30 13:05:40.275760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.275727 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:40.508448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:40.508410 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:41.862388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:41.862350 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:42.003902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:42.003868 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61452246-0bb8-4c65-bdbd-b2c3f53977cb" path="/var/lib/kubelet/pods/61452246-0bb8-4c65-bdbd-b2c3f53977cb/volumes" Oct 30 13:05:42.153004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:42.152963 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t"] Oct 30 13:05:42.153228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:42.153201 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" containerID="cri-o://48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610" gracePeriod=30 Oct 30 13:05:42.166878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:42.166814 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh"] Oct 30 13:05:42.167114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:42.167081 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" containerID="cri-o://9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a" gracePeriod=30 Oct 30 13:05:42.171798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:42.171770 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7"] Oct 30 13:05:42.171986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:42.171964 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" containerID="cri-o://e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac" gracePeriod=30 Oct 30 13:05:42.189950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:42.189893 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x"] Oct 30 13:05:42.190157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:42.190098 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" containerID="cri-o://ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037" gracePeriod=30 Oct 30 13:05:44.269695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:44.269664 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:45.938258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:45.938221 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:46.905037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:46.904994 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:47.512794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.512766 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 13:05:47.565550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.565509 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-data\") pod \"f43dd811-3ef2-4bf6-aab1-7db158180c51\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " Oct 30 13:05:47.565727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.565678 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-token\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-token\") pod \"f43dd811-3ef2-4bf6-aab1-7db158180c51\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " Oct 30 13:05:47.565727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.565718 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-envoy\") pod \"f43dd811-3ef2-4bf6-aab1-7db158180c51\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " Oct 30 13:05:47.565846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.565830 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43dd811-3ef2-4bf6-aab1-7db158180c51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43dd811-3ef2-4bf6-aab1-7db158180c51-istiod-ca-cert\") pod \"f43dd811-3ef2-4bf6-aab1-7db158180c51\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " Oct 30 13:05:47.565904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.565848 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-data" pod "f43dd811-3ef2-4bf6-aab1-7db158180c51" (UID: "f43dd811-3ef2-4bf6-aab1-7db158180c51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.565904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.565867 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-podinfo\") pod \"f43dd811-3ef2-4bf6-aab1-7db158180c51\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " Oct 30 13:05:47.566074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.565907 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-credential-socket\") pod \"f43dd811-3ef2-4bf6-aab1-7db158180c51\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " Oct 30 13:05:47.566074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.565936 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-certs\") pod \"f43dd811-3ef2-4bf6-aab1-7db158180c51\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " Oct 30 13:05:47.566074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.566048 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-kube-api-access-858vs\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-kube-api-access-858vs\") pod \"f43dd811-3ef2-4bf6-aab1-7db158180c51\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " Oct 30 13:05:47.566230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.566119 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-socket\") pod \"f43dd811-3ef2-4bf6-aab1-7db158180c51\" (UID: \"f43dd811-3ef2-4bf6-aab1-7db158180c51\") " Oct 30 13:05:47.566962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.566887 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43dd811-3ef2-4bf6-aab1-7db158180c51-istiod-ca-cert" pod "f43dd811-3ef2-4bf6-aab1-7db158180c51" (UID: "f43dd811-3ef2-4bf6-aab1-7db158180c51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:47.570983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.569137 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-socket" pod "f43dd811-3ef2-4bf6-aab1-7db158180c51" (UID: "f43dd811-3ef2-4bf6-aab1-7db158180c51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.570983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.570658 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-credential-socket" pod "f43dd811-3ef2-4bf6-aab1-7db158180c51" (UID: "f43dd811-3ef2-4bf6-aab1-7db158180c51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.570983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.570901 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-podinfo" pod "f43dd811-3ef2-4bf6-aab1-7db158180c51" (UID: "f43dd811-3ef2-4bf6-aab1-7db158180c51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:05:47.570983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.570957 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-certs" pod "f43dd811-3ef2-4bf6-aab1-7db158180c51" (UID: "f43dd811-3ef2-4bf6-aab1-7db158180c51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.572029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.571908 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.572029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.571938 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.572029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.571952 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43dd811-3ef2-4bf6-aab1-7db158180c51-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.572029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.571965 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.572029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.571977 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.572029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.571990 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.572529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.572506 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-kube-api-access-858vs" pod "f43dd811-3ef2-4bf6-aab1-7db158180c51" (UID: "f43dd811-3ef2-4bf6-aab1-7db158180c51"). InnerVolumeSpecName "kube-api-access-858vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:47.572616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.572598 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-envoy" pod "f43dd811-3ef2-4bf6-aab1-7db158180c51" (UID: "f43dd811-3ef2-4bf6-aab1-7db158180c51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.573962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.573938 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-token" pod "f43dd811-3ef2-4bf6-aab1-7db158180c51" (UID: "f43dd811-3ef2-4bf6-aab1-7db158180c51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:47.574519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.574499 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 13:05:47.576658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.576640 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 13:05:47.579901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.579882 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 13:05:47.672677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672613 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-envoy\") pod \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " Oct 30 13:05:47.672677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672672 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-socket\") pod \"53d01f34-7c20-47f0-8f05-d6c7155febcf\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " Oct 30 13:05:47.672928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672708 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-data\") pod \"a7d51232-79de-44ac-a895-4553f87ed4e8\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " Oct 30 13:05:47.672928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672737 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-credential-socket\") pod \"a7d51232-79de-44ac-a895-4553f87ed4e8\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " Oct 30 13:05:47.672928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672762 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-envoy\") pod \"53d01f34-7c20-47f0-8f05-d6c7155febcf\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " Oct 30 13:05:47.672928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672791 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-kube-api-access-r8kg9\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-kube-api-access-r8kg9\") pod \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " Oct 30 13:05:47.672928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672817 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-credential-socket\") pod \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " Oct 30 13:05:47.672928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672849 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-socket\") pod \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " Oct 30 13:05:47.672928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672881 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d51232-79de-44ac-a895-4553f87ed4e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d51232-79de-44ac-a895-4553f87ed4e8-istiod-ca-cert\") pod \"a7d51232-79de-44ac-a895-4553f87ed4e8\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672965 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-socket\") pod \"a7d51232-79de-44ac-a895-4553f87ed4e8\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.672991 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-token\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-token\") pod \"53d01f34-7c20-47f0-8f05-d6c7155febcf\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673048 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-token\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-token\") pod \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673076 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-data\") pod \"53d01f34-7c20-47f0-8f05-d6c7155febcf\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673131 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-envoy\") pod \"a7d51232-79de-44ac-a895-4553f87ed4e8\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673160 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-certs\") pod \"a7d51232-79de-44ac-a895-4553f87ed4e8\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673172 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-socket" pod "53d01f34-7c20-47f0-8f05-d6c7155febcf" (UID: "53d01f34-7c20-47f0-8f05-d6c7155febcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673201 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-podinfo\") pod \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673228 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53d01f34-7c20-47f0-8f05-d6c7155febcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d01f34-7c20-47f0-8f05-d6c7155febcf-istiod-ca-cert\") pod \"53d01f34-7c20-47f0-8f05-d6c7155febcf\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673369 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istiod-ca-cert\") pod \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673404 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-kube-api-access-2bhxg\") pod \"53d01f34-7c20-47f0-8f05-d6c7155febcf\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " Oct 30 13:05:47.673442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673434 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-credential-socket" pod "91f24ede-fb9a-4d3f-8a62-c8074270d5c9" (UID: "91f24ede-fb9a-4d3f-8a62-c8074270d5c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673467 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-socket" pod "91f24ede-fb9a-4d3f-8a62-c8074270d5c9" (UID: "91f24ede-fb9a-4d3f-8a62-c8074270d5c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673502 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-podinfo\") pod \"a7d51232-79de-44ac-a895-4553f87ed4e8\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673549 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-token\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-token\") pod \"a7d51232-79de-44ac-a895-4553f87ed4e8\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673586 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-podinfo\") pod \"53d01f34-7c20-47f0-8f05-d6c7155febcf\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673603 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-kube-api-access-zr2d4\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-kube-api-access-zr2d4\") pod \"a7d51232-79de-44ac-a895-4553f87ed4e8\" (UID: \"a7d51232-79de-44ac-a895-4553f87ed4e8\") " Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673631 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-certs\") pod \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673648 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-certs\") pod \"53d01f34-7c20-47f0-8f05-d6c7155febcf\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673675 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-data\") pod \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\" (UID: \"91f24ede-fb9a-4d3f-8a62-c8074270d5c9\") " Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673709 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-credential-socket\") pod \"53d01f34-7c20-47f0-8f05-d6c7155febcf\" (UID: \"53d01f34-7c20-47f0-8f05-d6c7155febcf\") " Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673759 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-data" pod "a7d51232-79de-44ac-a895-4553f87ed4e8" (UID: "a7d51232-79de-44ac-a895-4553f87ed4e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.673967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.673930 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-socket" pod "a7d51232-79de-44ac-a895-4553f87ed4e8" (UID: "a7d51232-79de-44ac-a895-4553f87ed4e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674192 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-credential-socket" pod "53d01f34-7c20-47f0-8f05-d6c7155febcf" (UID: "53d01f34-7c20-47f0-8f05-d6c7155febcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674222 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istiod-ca-cert" pod "91f24ede-fb9a-4d3f-8a62-c8074270d5c9" (UID: "91f24ede-fb9a-4d3f-8a62-c8074270d5c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674518 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674543 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674559 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43dd811-3ef2-4bf6-aab1-7db158180c51-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674571 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674585 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-858vs\" (UniqueName: \"kubernetes.io/projected/f43dd811-3ef2-4bf6-aab1-7db158180c51-kube-api-access-858vs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674598 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674611 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674624 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674637 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674648 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.675013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.674940 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-certs" pod "91f24ede-fb9a-4d3f-8a62-c8074270d5c9" (UID: "91f24ede-fb9a-4d3f-8a62-c8074270d5c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.675754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.675121 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-certs" pod "53d01f34-7c20-47f0-8f05-d6c7155febcf" (UID: "53d01f34-7c20-47f0-8f05-d6c7155febcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.675754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.675307 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-data" pod "91f24ede-fb9a-4d3f-8a62-c8074270d5c9" (UID: "91f24ede-fb9a-4d3f-8a62-c8074270d5c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.675754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.675495 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-data" pod "53d01f34-7c20-47f0-8f05-d6c7155febcf" (UID: "53d01f34-7c20-47f0-8f05-d6c7155febcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.676700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.676666 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-certs" pod "a7d51232-79de-44ac-a895-4553f87ed4e8" (UID: "a7d51232-79de-44ac-a895-4553f87ed4e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.676931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.676905 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d01f34-7c20-47f0-8f05-d6c7155febcf-istiod-ca-cert" pod "53d01f34-7c20-47f0-8f05-d6c7155febcf" (UID: "53d01f34-7c20-47f0-8f05-d6c7155febcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:47.677237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.677209 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d51232-79de-44ac-a895-4553f87ed4e8-istiod-ca-cert" pod "a7d51232-79de-44ac-a895-4553f87ed4e8" (UID: "a7d51232-79de-44ac-a895-4553f87ed4e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:47.677348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.677295 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-credential-socket" pod "a7d51232-79de-44ac-a895-4553f87ed4e8" (UID: "a7d51232-79de-44ac-a895-4553f87ed4e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.678691 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.678666 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-token" pod "53d01f34-7c20-47f0-8f05-d6c7155febcf" (UID: "53d01f34-7c20-47f0-8f05-d6c7155febcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:47.679422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.679252 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-kube-api-access-zr2d4" pod "a7d51232-79de-44ac-a895-4553f87ed4e8" (UID: "a7d51232-79de-44ac-a895-4553f87ed4e8"). InnerVolumeSpecName "kube-api-access-zr2d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:47.679422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.679292 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-podinfo" pod "53d01f34-7c20-47f0-8f05-d6c7155febcf" (UID: "53d01f34-7c20-47f0-8f05-d6c7155febcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:05:47.679422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.679326 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-envoy" pod "53d01f34-7c20-47f0-8f05-d6c7155febcf" (UID: "53d01f34-7c20-47f0-8f05-d6c7155febcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.679422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.679336 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-envoy" pod "91f24ede-fb9a-4d3f-8a62-c8074270d5c9" (UID: "91f24ede-fb9a-4d3f-8a62-c8074270d5c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.679422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.679331 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-kube-api-access-2bhxg" pod "53d01f34-7c20-47f0-8f05-d6c7155febcf" (UID: "53d01f34-7c20-47f0-8f05-d6c7155febcf"). InnerVolumeSpecName "kube-api-access-2bhxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:47.679422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.679368 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-kube-api-access-r8kg9" pod "91f24ede-fb9a-4d3f-8a62-c8074270d5c9" (UID: "91f24ede-fb9a-4d3f-8a62-c8074270d5c9"). InnerVolumeSpecName "kube-api-access-r8kg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:47.679422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.679390 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-podinfo" pod "a7d51232-79de-44ac-a895-4553f87ed4e8" (UID: "a7d51232-79de-44ac-a895-4553f87ed4e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:05:47.679744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.679489 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-token" pod "91f24ede-fb9a-4d3f-8a62-c8074270d5c9" (UID: "91f24ede-fb9a-4d3f-8a62-c8074270d5c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:47.679744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.679689 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-token" pod "a7d51232-79de-44ac-a895-4553f87ed4e8" (UID: "a7d51232-79de-44ac-a895-4553f87ed4e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:47.680147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.680128 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-envoy" pod "a7d51232-79de-44ac-a895-4553f87ed4e8" (UID: "a7d51232-79de-44ac-a895-4553f87ed4e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:47.680572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.680551 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-podinfo" pod "91f24ede-fb9a-4d3f-8a62-c8074270d5c9" (UID: "91f24ede-fb9a-4d3f-8a62-c8074270d5c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:05:47.777796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777760 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d51232-79de-44ac-a895-4553f87ed4e8-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.777796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777796 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777808 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777820 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777831 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777843 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777853 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777863 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d01f34-7c20-47f0-8f05-d6c7155febcf-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777877 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/53d01f34-7c20-47f0-8f05-d6c7155febcf-kube-api-access-2bhxg\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777890 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777901 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777912 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777923 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2d4\" (UniqueName: \"kubernetes.io/projected/a7d51232-79de-44ac-a895-4553f87ed4e8-kube-api-access-zr2d4\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777935 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777947 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777958 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777968 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777980 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d51232-79de-44ac-a895-4553f87ed4e8-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.777995 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d01f34-7c20-47f0-8f05-d6c7155febcf-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.778051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.778009 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8kg9\" (UniqueName: \"kubernetes.io/projected/91f24ede-fb9a-4d3f-8a62-c8074270d5c9-kube-api-access-r8kg9\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:47.917469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:47.917396 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:48.269425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.269338 5515 generic.go:356] "Generic (PLEG): container finished" podID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerID="e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac" exitCode=0 Oct 30 13:05:48.269425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.269386 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" Oct 30 13:05:48.269425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.269417 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" event={"ID":"f43dd811-3ef2-4bf6-aab1-7db158180c51","Type":"ContainerDied","Data":"e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac"} Oct 30 13:05:48.269724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.269470 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7" event={"ID":"f43dd811-3ef2-4bf6-aab1-7db158180c51","Type":"ContainerDied","Data":"644c4534ac768f584436bb21402631c91fd32c2c94f41f9090deb0f186fd799b"} Oct 30 13:05:48.269724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.269501 5515 scope.go:122] "RemoveContainer" containerID="e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac" Oct 30 13:05:48.270905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.270880 5515 generic.go:356] "Generic (PLEG): container finished" podID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerID="48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610" exitCode=0 Oct 30 13:05:48.271026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.270922 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" Oct 30 13:05:48.271026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.270962 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" event={"ID":"53d01f34-7c20-47f0-8f05-d6c7155febcf","Type":"ContainerDied","Data":"48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610"} Oct 30 13:05:48.271026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.271006 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t" event={"ID":"53d01f34-7c20-47f0-8f05-d6c7155febcf","Type":"ContainerDied","Data":"6a467516715c89b80e40fbf79e4d47d8a136022d7cb66fe528193f4454ad2537"} Oct 30 13:05:48.272545 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.272520 5515 generic.go:356] "Generic (PLEG): container finished" podID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerID="ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037" exitCode=0 Oct 30 13:05:48.272661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.272559 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" Oct 30 13:05:48.272661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.272592 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" event={"ID":"91f24ede-fb9a-4d3f-8a62-c8074270d5c9","Type":"ContainerDied","Data":"ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037"} Oct 30 13:05:48.272661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.272630 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x" event={"ID":"91f24ede-fb9a-4d3f-8a62-c8074270d5c9","Type":"ContainerDied","Data":"714f39b81cc60a279153ed8d346224a4dd9c1d05ecb74431fc5c001d9a59dcfd"} Oct 30 13:05:48.274538 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.274518 5515 generic.go:356] "Generic (PLEG): container finished" podID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerID="9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a" exitCode=0 Oct 30 13:05:48.274648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.274565 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" Oct 30 13:05:48.274648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.274595 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" event={"ID":"a7d51232-79de-44ac-a895-4553f87ed4e8","Type":"ContainerDied","Data":"9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a"} Oct 30 13:05:48.274648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.274623 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh" event={"ID":"a7d51232-79de-44ac-a895-4553f87ed4e8","Type":"ContainerDied","Data":"04defa35d964df2212450cead8978620a75fe091aec47df712285a26883386b0"} Oct 30 13:05:48.281588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.281569 5515 scope.go:122] "RemoveContainer" containerID="e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac" Oct 30 13:05:48.281860 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:48.281840 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac\": container with ID starting with e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac not found: ID does not exist" containerID="e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac" Oct 30 13:05:48.281938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.281873 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac"} err="failed to get container status \"e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac\": rpc error: code = NotFound desc = could not find container \"e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac\": container with ID starting with e4cc213ed8e78533783c1a0d8098fa0bc837e3ca69a32af978bb3da97661dcac not found: ID does not exist" Oct 30 13:05:48.281938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.281898 5515 scope.go:122] "RemoveContainer" containerID="48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610" Oct 30 13:05:48.291890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.291872 5515 scope.go:122] "RemoveContainer" containerID="48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610" Oct 30 13:05:48.292195 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:48.292170 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610\": container with ID starting with 48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610 not found: ID does not exist" containerID="48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610" Oct 30 13:05:48.292262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.292202 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610"} err="failed to get container status \"48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610\": rpc error: code = NotFound desc = could not find container \"48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610\": container with ID starting with 48dd0279a5e695b119d94a5be97a8a82fc1ef08b91d798006dff7784b59cb610 not found: ID does not exist" Oct 30 13:05:48.292262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.292221 5515 scope.go:122] "RemoveContainer" containerID="ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037" Oct 30 13:05:48.302199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.302030 5515 scope.go:122] "RemoveContainer" containerID="ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037" Oct 30 13:05:48.302485 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:48.302466 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037\": container with ID starting with ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037 not found: ID does not exist" containerID="ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037" Oct 30 13:05:48.302551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.302493 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037"} err="failed to get container status \"ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037\": rpc error: code = NotFound desc = could not find container \"ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037\": container with ID starting with ac7f0bc2a42696c96323fa856fb39af05907d77c57857c31571ad1c24a0fe037 not found: ID does not exist" Oct 30 13:05:48.302551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.302510 5515 scope.go:122] "RemoveContainer" containerID="9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a" Oct 30 13:05:48.312033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.312010 5515 scope.go:122] "RemoveContainer" containerID="442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d" Oct 30 13:05:48.325334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.325243 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7"] Oct 30 13:05:48.326185 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.326133 5515 scope.go:122] "RemoveContainer" containerID="9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a" Oct 30 13:05:48.326670 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:48.326642 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a\": container with ID starting with 9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a not found: ID does not exist" containerID="9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a" Oct 30 13:05:48.326797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.326681 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a"} err="failed to get container status \"9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a\": rpc error: code = NotFound desc = could not find container \"9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a\": container with ID starting with 9bd9d044e54973e91df902bbc9cdbb32b892055b27ff02f631040f64753e915a not found: ID does not exist" Oct 30 13:05:48.326797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.326706 5515 scope.go:122] "RemoveContainer" containerID="442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d" Oct 30 13:05:48.326994 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:48.326968 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d\": container with ID starting with 442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d not found: ID does not exist" containerID="442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d" Oct 30 13:05:48.327062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.327005 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d"} err="failed to get container status \"442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d\": rpc error: code = NotFound desc = could not find container \"442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d\": container with ID starting with 442217961bcac3a1357fd19b876601d1bc4e6d965cbc249cf6d57ec899e29e1d not found: ID does not exist" Oct 30 13:05:48.337667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.337635 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ds7v7"] Oct 30 13:05:48.400293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.396322 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t"] Oct 30 13:05:48.425355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.422789 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-9c98t"] Oct 30 13:05:48.466294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.461250 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:48.466294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.461665 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x"] Oct 30 13:05:48.488318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.486192 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-8s97x"] Oct 30 13:05:48.519649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.518135 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh"] Oct 30 13:05:48.522744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.522575 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-ndhdh"] Oct 30 13:05:48.648460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.648425 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b"] Oct 30 13:05:48.648728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.648701 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" containerID="cri-o://b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec" gracePeriod=30 Oct 30 13:05:48.803366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:48.802497 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:49.139740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:49.139701 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:49.729193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:49.726910 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:50.005198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:50.005120 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" path="/var/lib/kubelet/pods/53d01f34-7c20-47f0-8f05-d6c7155febcf/volumes" Oct 30 13:05:50.005687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:50.005671 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" path="/var/lib/kubelet/pods/91f24ede-fb9a-4d3f-8a62-c8074270d5c9/volumes" Oct 30 13:05:50.006119 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:50.006105 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" path="/var/lib/kubelet/pods/a7d51232-79de-44ac-a895-4553f87ed4e8/volumes" Oct 30 13:05:50.006777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:50.006760 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" path="/var/lib/kubelet/pods/f43dd811-3ef2-4bf6-aab1-7db158180c51/volumes" Oct 30 13:05:50.960075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:50.960031 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn"] Oct 30 13:05:51.001262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:51.001229 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8"] Oct 30 13:05:51.250166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:51.250071 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:51.589609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:51.589523 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:53.194239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:53.194203 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:53.532605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:53.532516 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n"] Oct 30 13:05:53.562861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:53.562833 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7"] Oct 30 13:05:53.865907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:53.865873 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:54.071594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.071533 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:05:54.216430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.216393 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:54.227587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.227548 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-socket\") pod \"55811f26-0cc4-42d3-b727-8f1c8f757450\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " Oct 30 13:05:54.227771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.227603 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-podinfo\") pod \"55811f26-0cc4-42d3-b727-8f1c8f757450\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " Oct 30 13:05:54.227771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.227663 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-certs\") pod \"55811f26-0cc4-42d3-b727-8f1c8f757450\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " Oct 30 13:05:54.227771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.227689 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-credential-socket\") pod \"55811f26-0cc4-42d3-b727-8f1c8f757450\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " Oct 30 13:05:54.227771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.227748 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-envoy\") pod \"55811f26-0cc4-42d3-b727-8f1c8f757450\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " Oct 30 13:05:54.228005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.227781 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-kube-api-access-rqlf5\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-kube-api-access-rqlf5\") pod \"55811f26-0cc4-42d3-b727-8f1c8f757450\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " Oct 30 13:05:54.228005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.227881 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-data\") pod \"55811f26-0cc4-42d3-b727-8f1c8f757450\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " Oct 30 13:05:54.228005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.227920 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55811f26-0cc4-42d3-b727-8f1c8f757450-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55811f26-0cc4-42d3-b727-8f1c8f757450-istiod-ca-cert\") pod \"55811f26-0cc4-42d3-b727-8f1c8f757450\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " Oct 30 13:05:54.228005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.227961 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-token\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-token\") pod \"55811f26-0cc4-42d3-b727-8f1c8f757450\" (UID: \"55811f26-0cc4-42d3-b727-8f1c8f757450\") " Oct 30 13:05:54.228230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.228004 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-certs" pod "55811f26-0cc4-42d3-b727-8f1c8f757450" (UID: "55811f26-0cc4-42d3-b727-8f1c8f757450"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:54.228230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.228056 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-credential-socket" pod "55811f26-0cc4-42d3-b727-8f1c8f757450" (UID: "55811f26-0cc4-42d3-b727-8f1c8f757450"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:54.228555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.228506 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-data" pod "55811f26-0cc4-42d3-b727-8f1c8f757450" (UID: "55811f26-0cc4-42d3-b727-8f1c8f757450"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:54.228658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.228621 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.228658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.228640 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.228658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.228653 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.228952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.228920 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-socket" pod "55811f26-0cc4-42d3-b727-8f1c8f757450" (UID: "55811f26-0cc4-42d3-b727-8f1c8f757450"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:54.231619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.230608 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-envoy" pod "55811f26-0cc4-42d3-b727-8f1c8f757450" (UID: "55811f26-0cc4-42d3-b727-8f1c8f757450"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:54.231619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.230777 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55811f26-0cc4-42d3-b727-8f1c8f757450-istiod-ca-cert" pod "55811f26-0cc4-42d3-b727-8f1c8f757450" (UID: "55811f26-0cc4-42d3-b727-8f1c8f757450"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:54.231917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.231896 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-podinfo" pod "55811f26-0cc4-42d3-b727-8f1c8f757450" (UID: "55811f26-0cc4-42d3-b727-8f1c8f757450"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:05:54.232485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.232461 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-token" pod "55811f26-0cc4-42d3-b727-8f1c8f757450" (UID: "55811f26-0cc4-42d3-b727-8f1c8f757450"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:54.232922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.232896 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-kube-api-access-rqlf5" pod "55811f26-0cc4-42d3-b727-8f1c8f757450" (UID: "55811f26-0cc4-42d3-b727-8f1c8f757450"). InnerVolumeSpecName "kube-api-access-rqlf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.329949 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.330058 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.330098 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.330134 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqlf5\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-kube-api-access-rqlf5\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.330148 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55811f26-0cc4-42d3-b727-8f1c8f757450-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.330161 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55811f26-0cc4-42d3-b727-8f1c8f757450-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.342557 5515 generic.go:356] "Generic (PLEG): container finished" podID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerID="b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec" exitCode=0 Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.342673 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" event={"ID":"55811f26-0cc4-42d3-b727-8f1c8f757450","Type":"ContainerDied","Data":"b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec"} Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.342726 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" event={"ID":"55811f26-0cc4-42d3-b727-8f1c8f757450","Type":"ContainerDied","Data":"c4beb4c83d81073022f2ba0b0eb434a7623541108ade2fe635d4126cf84ec7ec"} Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.342756 5515 scope.go:122] "RemoveContainer" containerID="b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec" Oct 30 13:05:54.355341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.342946 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b" Oct 30 13:05:54.414549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.413379 5515 scope.go:122] "RemoveContainer" containerID="626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925" Oct 30 13:05:54.459539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.458925 5515 scope.go:122] "RemoveContainer" containerID="b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec" Oct 30 13:05:54.459539 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:54.459415 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec\": container with ID starting with b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec not found: ID does not exist" containerID="b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec" Oct 30 13:05:54.459539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.459451 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec"} err="failed to get container status \"b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec\": rpc error: code = NotFound desc = could not find container \"b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec\": container with ID starting with b929029a292192794e5a753176300db1f4e6b38764f0449b436affef57ad3bec not found: ID does not exist" Oct 30 13:05:54.459539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.459478 5515 scope.go:122] "RemoveContainer" containerID="626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925" Oct 30 13:05:54.459960 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:54.459791 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925\": container with ID starting with 626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925 not found: ID does not exist" containerID="626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925" Oct 30 13:05:54.459960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.459819 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925"} err="failed to get container status \"626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925\": rpc error: code = NotFound desc = could not find container \"626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925\": container with ID starting with 626d653a350571a0206fdfe92dcc45689280537932ed8734e865a5842df3a925 not found: ID does not exist" Oct 30 13:05:54.509631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.509592 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b"] Oct 30 13:05:54.527321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:54.527281 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9368ce-openshift-default-54455b6995-6nc2b"] Oct 30 13:05:55.115128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:55.115094 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:55.961051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:55.960980 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" podUID="d8e90e60-fab3-49e7-9541-6c03658a7114" containerName="payload-processing" containerID="cri-o://f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8" gracePeriod=25 Oct 30 13:05:56.001590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.001553 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" podUID="32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da" containerName="payload-pre-processing" containerID="cri-o://b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9" gracePeriod=25 Oct 30 13:05:56.002472 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.002446 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" path="/var/lib/kubelet/pods/55811f26-0cc4-42d3-b727-8f1c8f757450/volumes" Oct 30 13:05:56.282989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.282961 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:05:56.314183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.314150 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:05:56.359682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.359650 5515 generic.go:356] "Generic (PLEG): container finished" podID="32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da" containerID="b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9" exitCode=0 Oct 30 13:05:56.359899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.359723 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" Oct 30 13:05:56.359899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.359746 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" event={"ID":"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da","Type":"ContainerDied","Data":"b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9"} Oct 30 13:05:56.359899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.359817 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8" event={"ID":"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da","Type":"ContainerDied","Data":"7453ddd848c1d82688822fcc871c37f8da21258159abfccc612668adfe9b6f72"} Oct 30 13:05:56.359899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.359851 5515 scope.go:122] "RemoveContainer" containerID="b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9" Oct 30 13:05:56.361601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.361579 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e90e60-fab3-49e7-9541-6c03658a7114-kube-api-access-fqkh9\" (UniqueName: \"kubernetes.io/projected/d8e90e60-fab3-49e7-9541-6c03658a7114-kube-api-access-fqkh9\") pod \"d8e90e60-fab3-49e7-9541-6c03658a7114\" (UID: \"d8e90e60-fab3-49e7-9541-6c03658a7114\") " Oct 30 13:05:56.361731 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.361630 5515 generic.go:356] "Generic (PLEG): container finished" podID="d8e90e60-fab3-49e7-9541-6c03658a7114" containerID="f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8" exitCode=0 Oct 30 13:05:56.361731 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.361707 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" Oct 30 13:05:56.361844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.361726 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" event={"ID":"d8e90e60-fab3-49e7-9541-6c03658a7114","Type":"ContainerDied","Data":"f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8"} Oct 30 13:05:56.361844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.361740 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e90e60-fab3-49e7-9541-6c03658a7114-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e90e60-fab3-49e7-9541-6c03658a7114-plugins-config-volume\") pod \"d8e90e60-fab3-49e7-9541-6c03658a7114\" (UID: \"d8e90e60-fab3-49e7-9541-6c03658a7114\") " Oct 30 13:05:56.361844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.361770 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn" event={"ID":"d8e90e60-fab3-49e7-9541-6c03658a7114","Type":"ContainerDied","Data":"363cd9db373884fe674ac46e3ad4fff09456dafa17a28f53225ffe905cce18bc"} Oct 30 13:05:56.362194 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.362158 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e90e60-fab3-49e7-9541-6c03658a7114-plugins-config-volume" pod "d8e90e60-fab3-49e7-9541-6c03658a7114" (UID: "d8e90e60-fab3-49e7-9541-6c03658a7114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:56.362424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.362406 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e90e60-fab3-49e7-9541-6c03658a7114-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.363791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.363768 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e90e60-fab3-49e7-9541-6c03658a7114-kube-api-access-fqkh9" pod "d8e90e60-fab3-49e7-9541-6c03658a7114" (UID: "d8e90e60-fab3-49e7-9541-6c03658a7114"). InnerVolumeSpecName "kube-api-access-fqkh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:56.378233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.378202 5515 scope.go:122] "RemoveContainer" containerID="b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9" Oct 30 13:05:56.378884 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:56.378860 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9\": container with ID starting with b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9 not found: ID does not exist" containerID="b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9" Oct 30 13:05:56.379000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.378891 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9"} err="failed to get container status \"b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9\": rpc error: code = NotFound desc = could not find container \"b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9\": container with ID starting with b919792d19bb2bd1a1c9b8ff3b8a0617161ac5b0c01765cf5bb5f9f818bd68a9 not found: ID does not exist" Oct 30 13:05:56.379000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.378909 5515 scope.go:122] "RemoveContainer" containerID="f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8" Oct 30 13:05:56.389864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.389838 5515 scope.go:122] "RemoveContainer" containerID="f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8" Oct 30 13:05:56.390224 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:56.390199 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8\": container with ID starting with f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8 not found: ID does not exist" containerID="f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8" Oct 30 13:05:56.390313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.390235 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8"} err="failed to get container status \"f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8\": rpc error: code = NotFound desc = could not find container \"f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8\": container with ID starting with f611c6511b29e0955199760cbb869c0030c8bb3b4399a8dc8d2f1e3bd5213da8 not found: ID does not exist" Oct 30 13:05:56.463548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.463507 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-kube-api-access-rn6t6\" (UniqueName: \"kubernetes.io/projected/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-kube-api-access-rn6t6\") pod \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\" (UID: \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\") " Oct 30 13:05:56.463729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.463677 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-plugins-config-volume\") pod \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\" (UID: \"32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da\") " Oct 30 13:05:56.464025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.464011 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqkh9\" (UniqueName: \"kubernetes.io/projected/d8e90e60-fab3-49e7-9541-6c03658a7114-kube-api-access-fqkh9\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.464097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.464051 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-plugins-config-volume" pod "32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da" (UID: "32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:56.465698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.465674 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-kube-api-access-rn6t6" pod "32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da" (UID: "32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da"). InnerVolumeSpecName "kube-api-access-rn6t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:56.565864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.565783 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn6t6\" (UniqueName: \"kubernetes.io/projected/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-kube-api-access-rn6t6\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.565864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.565812 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.731474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.731321 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn"] Oct 30 13:05:56.731474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.731381 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-68d9597b9b-n5cvn"] Oct 30 13:05:56.758983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.758940 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8"] Oct 30 13:05:56.763058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:56.763028 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-5d5885476c-gqsp8"] Oct 30 13:05:57.351660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:57.351566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:58.001856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:58.001820 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da" path="/var/lib/kubelet/pods/32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da/volumes" Oct 30 13:05:58.002259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:58.002245 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e90e60-fab3-49e7-9541-6c03658a7114" path="/var/lib/kubelet/pods/d8e90e60-fab3-49e7-9541-6c03658a7114/volumes" Oct 30 13:05:58.499162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:58.499125 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88"] Oct 30 13:05:58.533017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:58.532979 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn"] Oct 30 13:05:58.533205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:58.533001 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" podUID="267e14eb-68e6-49a3-bbf4-a500f9a6e66e" containerName="payload-processing" containerID="cri-o://015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5" gracePeriod=25 Oct 30 13:05:58.564078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:58.564031 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" podUID="bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7" containerName="payload-pre-processing" containerID="cri-o://db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34" gracePeriod=25 Oct 30 13:05:58.762359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:58.761479 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:05:58.764279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:58.764238 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:58.851922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:58.851854 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:59.010372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.010155 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-plugins-config-volume\") pod \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\" (UID: \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\") " Oct 30 13:05:59.010372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.010247 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-kube-api-access-4d2hr\" (UniqueName: \"kubernetes.io/projected/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-kube-api-access-4d2hr\") pod \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\" (UID: \"267e14eb-68e6-49a3-bbf4-a500f9a6e66e\") " Oct 30 13:05:59.055016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.054969 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-kube-api-access-4d2hr" pod "267e14eb-68e6-49a3-bbf4-a500f9a6e66e" (UID: "267e14eb-68e6-49a3-bbf4-a500f9a6e66e"). InnerVolumeSpecName "kube-api-access-4d2hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:59.055663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.055633 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:59.063323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.059809 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-plugins-config-volume" pod "267e14eb-68e6-49a3-bbf4-a500f9a6e66e" (UID: "267e14eb-68e6-49a3-bbf4-a500f9a6e66e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:59.076480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.076457 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:05:59.083055 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:59.083024 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:05:59.112667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.112632 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:59.112667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.112656 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d2hr\" (UniqueName: \"kubernetes.io/projected/267e14eb-68e6-49a3-bbf4-a500f9a6e66e-kube-api-access-4d2hr\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:59.213755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.213723 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-kube-api-access-bv8ls\" (UniqueName: \"kubernetes.io/projected/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-kube-api-access-bv8ls\") pod \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\" (UID: \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\") " Oct 30 13:05:59.213938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.213830 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-plugins-config-volume\") pod \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\" (UID: \"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7\") " Oct 30 13:05:59.214287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.214246 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-plugins-config-volume" pod "bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7" (UID: "bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:59.215935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.215912 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-kube-api-access-bv8ls" pod "bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7" (UID: "bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7"). InnerVolumeSpecName "kube-api-access-bv8ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:59.315071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.314992 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv8ls\" (UniqueName: \"kubernetes.io/projected/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-kube-api-access-bv8ls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:59.315071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.315019 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:05:59.441333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.434415 5515 generic.go:356] "Generic (PLEG): container finished" podID="267e14eb-68e6-49a3-bbf4-a500f9a6e66e" containerID="015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5" exitCode=0 Oct 30 13:05:59.441333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.434689 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" event={"ID":"267e14eb-68e6-49a3-bbf4-a500f9a6e66e","Type":"ContainerDied","Data":"015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5"} Oct 30 13:05:59.441333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.434767 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" event={"ID":"267e14eb-68e6-49a3-bbf4-a500f9a6e66e","Type":"ContainerDied","Data":"d443d363864aa6acc91a19233a4bb82c5511fb9ba5b578c210241809fe5def16"} Oct 30 13:05:59.441333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.434799 5515 scope.go:122] "RemoveContainer" containerID="015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5" Oct 30 13:05:59.441333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.434994 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n" Oct 30 13:05:59.465875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.465828 5515 generic.go:356] "Generic (PLEG): container finished" podID="bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7" containerID="db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34" exitCode=0 Oct 30 13:05:59.466537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.466188 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" event={"ID":"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7","Type":"ContainerDied","Data":"db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34"} Oct 30 13:05:59.467118 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.466914 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" event={"ID":"bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7","Type":"ContainerDied","Data":"5a449f160244a2c69cad1dee15c8620acd3a3c5efbd5f7fa4766d5ed174de95e"} Oct 30 13:05:59.467395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.466319 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7" Oct 30 13:05:59.525515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.523189 5515 scope.go:122] "RemoveContainer" containerID="015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5" Oct 30 13:05:59.525515 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:59.524827 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5\": container with ID starting with 015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5 not found: ID does not exist" containerID="015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5" Oct 30 13:05:59.525515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.524863 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5"} err="failed to get container status \"015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5\": rpc error: code = NotFound desc = could not find container \"015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5\": container with ID starting with 015c77678687b64940dd3e299451b6ecb120d5e4c12267161b77922f81080ff5 not found: ID does not exist" Oct 30 13:05:59.525515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.524886 5515 scope.go:122] "RemoveContainer" containerID="db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34" Oct 30 13:05:59.539369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.539253 5515 scope.go:122] "RemoveContainer" containerID="db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34" Oct 30 13:05:59.539762 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:05:59.539736 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34\": container with ID starting with db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34 not found: ID does not exist" containerID="db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34" Oct 30 13:05:59.539871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.539770 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34"} err="failed to get container status \"db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34\": rpc error: code = NotFound desc = could not find container \"db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34\": container with ID starting with db08d2511ac0a14ac2287f392987494899432097c5d2ff9a0d856f96be3ddc34 not found: ID does not exist" Oct 30 13:05:59.547869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.547836 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n"] Oct 30 13:05:59.559976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.559937 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-5fbb7c76b6-5mw9n"] Oct 30 13:05:59.587505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.587420 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7"] Oct 30 13:05:59.589921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:05:59.589900 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-747bdf6795-lhvr7"] Oct 30 13:06:00.002699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.002663 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267e14eb-68e6-49a3-bbf4-a500f9a6e66e" path="/var/lib/kubelet/pods/267e14eb-68e6-49a3-bbf4-a500f9a6e66e/volumes" Oct 30 13:06:00.003228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.003205 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7" path="/var/lib/kubelet/pods/bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7/volumes" Oct 30 13:06:00.191429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.191396 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j"] Oct 30 13:06:00.191990 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.191931 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" podUID="c15fd225-5060-4b06-95ce-682e90d25cdc" containerName="maas-api" containerID="cri-o://1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712" gracePeriod=30 Oct 30 13:06:00.396062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.396020 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:00.486366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.485240 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:06:00.503420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.503387 5515 generic.go:356] "Generic (PLEG): container finished" podID="c15fd225-5060-4b06-95ce-682e90d25cdc" containerID="1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712" exitCode=0 Oct 30 13:06:00.503569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.503512 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" Oct 30 13:06:00.503569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.503528 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" event={"ID":"c15fd225-5060-4b06-95ce-682e90d25cdc","Type":"ContainerDied","Data":"1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712"} Oct 30 13:06:00.503687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.503668 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j" event={"ID":"c15fd225-5060-4b06-95ce-682e90d25cdc","Type":"ContainerDied","Data":"79addd98a19a3ca664854f747990b39a3e5aae1ba2f5ea1c07e9a8efc2e593d8"} Oct 30 13:06:00.503736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.503688 5515 scope.go:122] "RemoveContainer" containerID="1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712" Oct 30 13:06:00.515826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.515800 5515 scope.go:122] "RemoveContainer" containerID="1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712" Oct 30 13:06:00.516137 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:00.516115 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712\": container with ID starting with 1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712 not found: ID does not exist" containerID="1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712" Oct 30 13:06:00.516224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.516150 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712"} err="failed to get container status \"1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712\": rpc error: code = NotFound desc = could not find container \"1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712\": container with ID starting with 1bbed4ec5a238c99e7d1f91fdfae5ca05e2fcff7d74ab848d47e734b7a5c0712 not found: ID does not exist" Oct 30 13:06:00.654232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.654192 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-maas-api-tls\") pod \"c15fd225-5060-4b06-95ce-682e90d25cdc\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " Oct 30 13:06:00.654780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.654322 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15fd225-5060-4b06-95ce-682e90d25cdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15fd225-5060-4b06-95ce-682e90d25cdc-openshift-service-ca\") pod \"c15fd225-5060-4b06-95ce-682e90d25cdc\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " Oct 30 13:06:00.654780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.654412 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-metrics-certs\") pod \"c15fd225-5060-4b06-95ce-682e90d25cdc\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " Oct 30 13:06:00.654780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.654462 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15fd225-5060-4b06-95ce-682e90d25cdc-kube-api-access-428pf\" (UniqueName: \"kubernetes.io/projected/c15fd225-5060-4b06-95ce-682e90d25cdc-kube-api-access-428pf\") pod \"c15fd225-5060-4b06-95ce-682e90d25cdc\" (UID: \"c15fd225-5060-4b06-95ce-682e90d25cdc\") " Oct 30 13:06:00.654780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.654765 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15fd225-5060-4b06-95ce-682e90d25cdc-openshift-service-ca" pod "c15fd225-5060-4b06-95ce-682e90d25cdc" (UID: "c15fd225-5060-4b06-95ce-682e90d25cdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:00.655057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.655031 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c15fd225-5060-4b06-95ce-682e90d25cdc-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:00.656594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.656562 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15fd225-5060-4b06-95ce-682e90d25cdc-kube-api-access-428pf" pod "c15fd225-5060-4b06-95ce-682e90d25cdc" (UID: "c15fd225-5060-4b06-95ce-682e90d25cdc"). InnerVolumeSpecName "kube-api-access-428pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:00.656701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.656600 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-maas-api-tls" pod "c15fd225-5060-4b06-95ce-682e90d25cdc" (UID: "c15fd225-5060-4b06-95ce-682e90d25cdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:00.656701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.656643 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-metrics-certs" pod "c15fd225-5060-4b06-95ce-682e90d25cdc" (UID: "c15fd225-5060-4b06-95ce-682e90d25cdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:00.730431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.728185 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:00.772294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.759233 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:00.772294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.759319 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-428pf\" (UniqueName: \"kubernetes.io/projected/c15fd225-5060-4b06-95ce-682e90d25cdc-kube-api-access-428pf\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:00.772294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.759338 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c15fd225-5060-4b06-95ce-682e90d25cdc-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:00.938584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.938467 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j"] Oct 30 13:06:00.943617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:00.943576 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f0f50-779b7f9794-m686j"] Oct 30 13:06:01.018848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:01.018808 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:01.257635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:01.257555 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:01.758511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:01.758482 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:02.001992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:02.001950 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15fd225-5060-4b06-95ce-682e90d25cdc" path="/var/lib/kubelet/pods/c15fd225-5060-4b06-95ce-682e90d25cdc/volumes" Oct 30 13:06:02.947943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:02.947907 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:03.499909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.499872 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" podUID="48b0c77f-f12b-4455-86d6-96927ed354fa" containerName="payload-processing" containerID="cri-o://e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29" gracePeriod=25 Oct 30 13:06:03.534031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.533985 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" podUID="d554adfd-5c59-4989-9061-366702999398" containerName="payload-pre-processing" containerID="cri-o://7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2" gracePeriod=25 Oct 30 13:06:03.818021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.817988 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:06:03.831651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.831620 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b0c77f-f12b-4455-86d6-96927ed354fa-kube-api-access-2rc7w\" (UniqueName: \"kubernetes.io/projected/48b0c77f-f12b-4455-86d6-96927ed354fa-kube-api-access-2rc7w\") pod \"48b0c77f-f12b-4455-86d6-96927ed354fa\" (UID: \"48b0c77f-f12b-4455-86d6-96927ed354fa\") " Oct 30 13:06:03.831807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.831745 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b0c77f-f12b-4455-86d6-96927ed354fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b0c77f-f12b-4455-86d6-96927ed354fa-plugins-config-volume\") pod \"48b0c77f-f12b-4455-86d6-96927ed354fa\" (UID: \"48b0c77f-f12b-4455-86d6-96927ed354fa\") " Oct 30 13:06:03.832110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.832084 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b0c77f-f12b-4455-86d6-96927ed354fa-plugins-config-volume" pod "48b0c77f-f12b-4455-86d6-96927ed354fa" (UID: "48b0c77f-f12b-4455-86d6-96927ed354fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:03.833911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.833884 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b0c77f-f12b-4455-86d6-96927ed354fa-kube-api-access-2rc7w" pod "48b0c77f-f12b-4455-86d6-96927ed354fa" (UID: "48b0c77f-f12b-4455-86d6-96927ed354fa"). InnerVolumeSpecName "kube-api-access-2rc7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:03.855133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.855109 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:06:03.933556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.933507 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d554adfd-5c59-4989-9061-366702999398-kube-api-access-4qghh\" (UniqueName: \"kubernetes.io/projected/d554adfd-5c59-4989-9061-366702999398-kube-api-access-4qghh\") pod \"d554adfd-5c59-4989-9061-366702999398\" (UID: \"d554adfd-5c59-4989-9061-366702999398\") " Oct 30 13:06:03.933761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.933576 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d554adfd-5c59-4989-9061-366702999398-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554adfd-5c59-4989-9061-366702999398-plugins-config-volume\") pod \"d554adfd-5c59-4989-9061-366702999398\" (UID: \"d554adfd-5c59-4989-9061-366702999398\") " Oct 30 13:06:03.934235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.934202 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rc7w\" (UniqueName: \"kubernetes.io/projected/48b0c77f-f12b-4455-86d6-96927ed354fa-kube-api-access-2rc7w\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:03.934384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.934238 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48b0c77f-f12b-4455-86d6-96927ed354fa-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:03.934384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.934321 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d554adfd-5c59-4989-9061-366702999398-plugins-config-volume" pod "d554adfd-5c59-4989-9061-366702999398" (UID: "d554adfd-5c59-4989-9061-366702999398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:03.936651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:03.936622 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d554adfd-5c59-4989-9061-366702999398-kube-api-access-4qghh" pod "d554adfd-5c59-4989-9061-366702999398" (UID: "d554adfd-5c59-4989-9061-366702999398"). InnerVolumeSpecName "kube-api-access-4qghh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:04.036365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.036279 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qghh\" (UniqueName: \"kubernetes.io/projected/d554adfd-5c59-4989-9061-366702999398-kube-api-access-4qghh\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.036365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.036311 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d554adfd-5c59-4989-9061-366702999398-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:04.534407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.534376 5515 generic.go:356] "Generic (PLEG): container finished" podID="d554adfd-5c59-4989-9061-366702999398" containerID="7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2" exitCode=0 Oct 30 13:06:04.534605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.534420 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" Oct 30 13:06:04.534605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.534449 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" event={"ID":"d554adfd-5c59-4989-9061-366702999398","Type":"ContainerDied","Data":"7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2"} Oct 30 13:06:04.534605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.534514 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn" event={"ID":"d554adfd-5c59-4989-9061-366702999398","Type":"ContainerDied","Data":"7597b988732800ddf68a90c6f80fb4f502268eb98a58b4a4167926c054a60ff5"} Oct 30 13:06:04.534605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.534546 5515 scope.go:122] "RemoveContainer" containerID="7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2" Oct 30 13:06:04.536002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.535976 5515 generic.go:356] "Generic (PLEG): container finished" podID="48b0c77f-f12b-4455-86d6-96927ed354fa" containerID="e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29" exitCode=0 Oct 30 13:06:04.536111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.536021 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" Oct 30 13:06:04.536111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.536067 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" event={"ID":"48b0c77f-f12b-4455-86d6-96927ed354fa","Type":"ContainerDied","Data":"e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29"} Oct 30 13:06:04.536207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.536137 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88" event={"ID":"48b0c77f-f12b-4455-86d6-96927ed354fa","Type":"ContainerDied","Data":"fb279acdb41147547c780c6ab75cf681c345cd495a4c5762de675e6d45453725"} Oct 30 13:06:04.544437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.544417 5515 scope.go:122] "RemoveContainer" containerID="7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2" Oct 30 13:06:04.544722 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:04.544703 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2\": container with ID starting with 7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2 not found: ID does not exist" containerID="7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2" Oct 30 13:06:04.544764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.544732 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2"} err="failed to get container status \"7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2\": rpc error: code = NotFound desc = could not find container \"7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2\": container with ID starting with 7377fea5e1139a58ca9a38aea4bca0fd91909443f19d5edd86b4144d40a482d2 not found: ID does not exist" Oct 30 13:06:04.544764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.544751 5515 scope.go:122] "RemoveContainer" containerID="e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29" Oct 30 13:06:04.554223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.554192 5515 scope.go:122] "RemoveContainer" containerID="e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29" Oct 30 13:06:04.554586 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:04.554562 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29\": container with ID starting with e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29 not found: ID does not exist" containerID="e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29" Oct 30 13:06:04.554649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.554595 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29"} err="failed to get container status \"e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29\": rpc error: code = NotFound desc = could not find container \"e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29\": container with ID starting with e9f889eb2033a38cb631da6253a11e2fe594a18d2c19ef7767963ff5c7401c29 not found: ID does not exist" Oct 30 13:06:04.587321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.587284 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88"] Oct 30 13:06:04.599663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.599629 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3a236d-85d875b7fb-sbg88"] Oct 30 13:06:04.633951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.633917 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn"] Oct 30 13:06:04.638260 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:04.638227 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a236d-75ff5cb6bc-2smdn"] Oct 30 13:06:05.038181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:05.038018 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:06.002764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.002731 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b0c77f-f12b-4455-86d6-96927ed354fa" path="/var/lib/kubelet/pods/48b0c77f-f12b-4455-86d6-96927ed354fa/volumes" Oct 30 13:06:06.003208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.003193 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d554adfd-5c59-4989-9061-366702999398" path="/var/lib/kubelet/pods/d554adfd-5c59-4989-9061-366702999398/volumes" Oct 30 13:06:06.199987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.199956 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:06.457292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.446104 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:06.479792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.479751 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs"] Oct 30 13:06:06.490761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.490729 5515 cpu_manager.go:414] "containerMap: removing container" podUID="61452246-0bb8-4c65-bdbd-b2c3f53977cb" containerName="maas-api" Oct 30 13:06:06.490993 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.490973 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61452246-0bb8-4c65-bdbd-b2c3f53977cb" containerName="maas-api" Oct 30 13:06:06.491164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.491149 5515 cpu_manager.go:414] "containerMap: removing container" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" Oct 30 13:06:06.491296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.491277 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" Oct 30 13:06:06.491453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.491440 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d554adfd-5c59-4989-9061-366702999398" containerName="payload-pre-processing" Oct 30 13:06:06.491564 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.491551 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d554adfd-5c59-4989-9061-366702999398" containerName="payload-pre-processing" Oct 30 13:06:06.491669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.491659 5515 cpu_manager.go:414] "containerMap: removing container" podUID="48b0c77f-f12b-4455-86d6-96927ed354fa" containerName="payload-processing" Oct 30 13:06:06.491753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.491743 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b0c77f-f12b-4455-86d6-96927ed354fa" containerName="payload-processing" Oct 30 13:06:06.491853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.491843 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" Oct 30 13:06:06.491938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.491928 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" Oct 30 13:06:06.492030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492020 5515 cpu_manager.go:414] "containerMap: removing container" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" Oct 30 13:06:06.492123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492113 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" Oct 30 13:06:06.492217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492207 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c15fd225-5060-4b06-95ce-682e90d25cdc" containerName="maas-api" Oct 30 13:06:06.492332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492321 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15fd225-5060-4b06-95ce-682e90d25cdc" containerName="maas-api" Oct 30 13:06:06.492425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492415 5515 cpu_manager.go:414] "containerMap: removing container" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" Oct 30 13:06:06.492514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492503 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" Oct 30 13:06:06.492622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492611 5515 cpu_manager.go:414] "containerMap: removing container" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" Oct 30 13:06:06.492734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492715 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" Oct 30 13:06:06.492872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492859 5515 cpu_manager.go:414] "containerMap: removing container" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" Oct 30 13:06:06.492975 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.492964 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" Oct 30 13:06:06.493073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.493063 5515 cpu_manager.go:414] "containerMap: removing container" podUID="32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da" containerName="payload-pre-processing" Oct 30 13:06:06.493157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.493146 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da" containerName="payload-pre-processing" Oct 30 13:06:06.493300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.493290 5515 cpu_manager.go:414] "containerMap: removing container" podUID="267e14eb-68e6-49a3-bbf4-a500f9a6e66e" containerName="payload-processing" Oct 30 13:06:06.493407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.493395 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267e14eb-68e6-49a3-bbf4-a500f9a6e66e" containerName="payload-processing" Oct 30 13:06:06.493539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.493528 5515 cpu_manager.go:414] "containerMap: removing container" podUID="bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7" containerName="payload-pre-processing" Oct 30 13:06:06.493635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.493623 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7" containerName="payload-pre-processing" Oct 30 13:06:06.493761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.493748 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d8e90e60-fab3-49e7-9541-6c03658a7114" containerName="payload-processing" Oct 30 13:06:06.493890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.493849 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e90e60-fab3-49e7-9541-6c03658a7114" containerName="payload-processing" Oct 30 13:06:06.494256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.494240 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="267e14eb-68e6-49a3-bbf4-a500f9a6e66e" containerName="payload-processing" Oct 30 13:06:06.494405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.494391 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd55a2e2-aaaf-4a2f-b99d-5254c25c77f7" containerName="payload-pre-processing" Oct 30 13:06:06.494524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.494512 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" Oct 30 13:06:06.494630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.494611 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e90e60-fab3-49e7-9541-6c03658a7114" containerName="payload-processing" Oct 30 13:06:06.494738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.494726 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="61452246-0bb8-4c65-bdbd-b2c3f53977cb" containerName="maas-api" Oct 30 13:06:06.494837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.494827 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d01f34-7c20-47f0-8f05-d6c7155febcf" containerName="istio-proxy" Oct 30 13:06:06.496637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.496621 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d554adfd-5c59-4989-9061-366702999398" containerName="payload-pre-processing" Oct 30 13:06:06.496779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.496766 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b0c77f-f12b-4455-86d6-96927ed354fa" containerName="payload-processing" Oct 30 13:06:06.496885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.496874 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43dd811-3ef2-4bf6-aab1-7db158180c51" containerName="istio-proxy" Oct 30 13:06:06.496992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.496982 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" Oct 30 13:06:06.497084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.497075 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15fd225-5060-4b06-95ce-682e90d25cdc" containerName="maas-api" Oct 30 13:06:06.497178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.497168 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="55811f26-0cc4-42d3-b727-8f1c8f757450" containerName="istio-proxy" Oct 30 13:06:06.497279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.497256 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f24ede-fb9a-4d3f-8a62-c8074270d5c9" containerName="istio-proxy" Oct 30 13:06:06.497387 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.497376 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fce8bb-0b1f-4e49-9f0a-43fa3f7a17da" containerName="payload-pre-processing" Oct 30 13:06:06.533104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.531614 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs"] Oct 30 13:06:06.533104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.531749 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:06.548847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.548813 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fa855c\"" Oct 30 13:06:06.560076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.560031 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fa855c-dockercfg-vj8rj\"" Oct 30 13:06:06.586584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.586547 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0263e3e3-ec51-4454-84bd-5634d45ba500-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs\" (UID: \"0263e3e3-ec51-4454-84bd-5634d45ba500\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:06.586738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.586606 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d2km\" (UniqueName: \"kubernetes.io/projected/0263e3e3-ec51-4454-84bd-5634d45ba500-kube-api-access-4d2km\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs\" (UID: \"0263e3e3-ec51-4454-84bd-5634d45ba500\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:06.671793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.671756 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9"] Oct 30 13:06:06.672447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.672430 5515 cpu_manager.go:414] "containerMap: removing container" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" Oct 30 13:06:06.672501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.672448 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" Oct 30 13:06:06.672544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.672535 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d51232-79de-44ac-a895-4553f87ed4e8" containerName="istio-proxy" Oct 30 13:06:06.676083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.676065 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:06.688746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.688714 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0263e3e3-ec51-4454-84bd-5634d45ba500-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs\" (UID: \"0263e3e3-ec51-4454-84bd-5634d45ba500\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:06.688919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.688762 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d2km\" (UniqueName: \"kubernetes.io/projected/0263e3e3-ec51-4454-84bd-5634d45ba500-kube-api-access-4d2km\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs\" (UID: \"0263e3e3-ec51-4454-84bd-5634d45ba500\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:06.689478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.689456 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0263e3e3-ec51-4454-84bd-5634d45ba500-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs\" (UID: \"0263e3e3-ec51-4454-84bd-5634d45ba500\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:06.696330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.696300 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9"] Oct 30 13:06:06.713570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.713499 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d2km\" (UniqueName: \"kubernetes.io/projected/0263e3e3-ec51-4454-84bd-5634d45ba500-kube-api-access-4d2km\") pod \"payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs\" (UID: \"0263e3e3-ec51-4454-84bd-5634d45ba500\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:06.790186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.790143 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dd2400-473f-4788-942c-f0ed36def00c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9\" (UID: \"19dd2400-473f-4788-942c-f0ed36def00c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:06.790468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.790426 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9l7\" (UniqueName: \"kubernetes.io/projected/19dd2400-473f-4788-942c-f0ed36def00c-kube-api-access-sg9l7\") pod \"payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9\" (UID: \"19dd2400-473f-4788-942c-f0ed36def00c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:06.854338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.854297 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:06.891238 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.891197 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9l7\" (UniqueName: \"kubernetes.io/projected/19dd2400-473f-4788-942c-f0ed36def00c-kube-api-access-sg9l7\") pod \"payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9\" (UID: \"19dd2400-473f-4788-942c-f0ed36def00c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:06.891628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.891327 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dd2400-473f-4788-942c-f0ed36def00c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9\" (UID: \"19dd2400-473f-4788-942c-f0ed36def00c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:06.891956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.891935 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dd2400-473f-4788-942c-f0ed36def00c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9\" (UID: \"19dd2400-473f-4788-942c-f0ed36def00c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:06.915437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.915403 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9l7\" (UniqueName: \"kubernetes.io/projected/19dd2400-473f-4788-942c-f0ed36def00c-kube-api-access-sg9l7\") pod \"payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9\" (UID: \"19dd2400-473f-4788-942c-f0ed36def00c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:06.991105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:06.991002 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:07.182758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.182672 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs"] Oct 30 13:06:07.205360 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:06:07.205310 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0263e3e3_ec51_4454_84bd_5634d45ba500.slice/crio-f619109b0643aaf5c593cf3ac725e07a7ce96ef7f8eb0d087003c120dacd9da1 WatchSource:0}: Error finding container f619109b0643aaf5c593cf3ac725e07a7ce96ef7f8eb0d087003c120dacd9da1: Status 404 returned error can't find the container with id f619109b0643aaf5c593cf3ac725e07a7ce96ef7f8eb0d087003c120dacd9da1 Oct 30 13:06:07.209653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.209612 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:07.296083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.296053 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9"] Oct 30 13:06:07.562698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.562603 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" event={"ID":"19dd2400-473f-4788-942c-f0ed36def00c","Type":"ContainerStarted","Data":"871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9"} Oct 30 13:06:07.562698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.562672 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:07.562698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.562692 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" event={"ID":"19dd2400-473f-4788-942c-f0ed36def00c","Type":"ContainerStarted","Data":"51b788d0067ef530bc0fd6950dcc2d6dca77f111ec15fa36c077b540f5f40c8b"} Oct 30 13:06:07.564346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.564314 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" event={"ID":"0263e3e3-ec51-4454-84bd-5634d45ba500","Type":"ContainerStarted","Data":"8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384"} Oct 30 13:06:07.564489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.564371 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" event={"ID":"0263e3e3-ec51-4454-84bd-5634d45ba500","Type":"ContainerStarted","Data":"f619109b0643aaf5c593cf3ac725e07a7ce96ef7f8eb0d087003c120dacd9da1"} Oct 30 13:06:07.564489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.564420 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:07.608906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.608836 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" podStartSLOduration=1.6088165939999999 podStartE2EDuration="1.608816594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:07.60395171 +0000 UTC m=+1985.730752610" watchObservedRunningTime="2026-10-30 13:06:07.608816594 +0000 UTC m=+1985.735617495" Oct 30 13:06:07.691655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:07.691591 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" podStartSLOduration=1.691577812 podStartE2EDuration="1.691577812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:07.690339302 +0000 UTC m=+1985.817140197" watchObservedRunningTime="2026-10-30 13:06:07.691577812 +0000 UTC m=+1985.818378710" Oct 30 13:06:08.698300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:08.698232 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:08.909678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:08.909635 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t"] Oct 30 13:06:08.909956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:08.909925 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" containerID="cri-o://c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30" gracePeriod=30 Oct 30 13:06:08.988308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:08.988204 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:09.309390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:09.309298 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:09.973505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:09.973473 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.252678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.251907 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.271320 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.270605 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k"] Oct 30 13:06:10.276139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.276039 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:10.295134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.294976 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k"] Oct 30 13:06:10.300447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.300415 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f9c553\"" Oct 30 13:06:10.302786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.301403 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f9c553-dockercfg-b7rfb\"" Oct 30 13:06:10.377131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.377100 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d99a3b-3579-431d-8bd8-958afee0a9a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k\" (UID: \"86d99a3b-3579-431d-8bd8-958afee0a9a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:10.377316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.377214 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89m9l\" (UniqueName: \"kubernetes.io/projected/86d99a3b-3579-431d-8bd8-958afee0a9a0-kube-api-access-89m9l\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k\" (UID: \"86d99a3b-3579-431d-8bd8-958afee0a9a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:10.478423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.478382 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89m9l\" (UniqueName: \"kubernetes.io/projected/86d99a3b-3579-431d-8bd8-958afee0a9a0-kube-api-access-89m9l\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k\" (UID: \"86d99a3b-3579-431d-8bd8-958afee0a9a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:10.478655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.478634 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d99a3b-3579-431d-8bd8-958afee0a9a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k\" (UID: \"86d99a3b-3579-431d-8bd8-958afee0a9a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:10.479636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.479608 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d99a3b-3579-431d-8bd8-958afee0a9a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k\" (UID: \"86d99a3b-3579-431d-8bd8-958afee0a9a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:10.501692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.501657 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.504216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.504141 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89m9l\" (UniqueName: \"kubernetes.io/projected/86d99a3b-3579-431d-8bd8-958afee0a9a0-kube-api-access-89m9l\") pod \"payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k\" (UID: \"86d99a3b-3579-431d-8bd8-958afee0a9a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:10.612101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.612062 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:10.917088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:10.917043 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k"] Oct 30 13:06:10.917841 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:06:10.917811 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d99a3b_3579_431d_8bd8_958afee0a9a0.slice/crio-92977de93d76b3bc7a6bbb39babf498b91d0de8c3dc3d0ba77904d36bd5933bd WatchSource:0}: Error finding container 92977de93d76b3bc7a6bbb39babf498b91d0de8c3dc3d0ba77904d36bd5933bd: Status 404 returned error can't find the container with id 92977de93d76b3bc7a6bbb39babf498b91d0de8c3dc3d0ba77904d36bd5933bd Oct 30 13:06:11.073053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:11.073017 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:11.349055 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:11.348962 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:11.592447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:11.592401 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" event={"ID":"86d99a3b-3579-431d-8bd8-958afee0a9a0","Type":"ContainerStarted","Data":"9b96e0f99eae9ae9ff7c66f178adfd0ae9e6edba3192b29ee47ab9df7894cd77"} Oct 30 13:06:11.592608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:11.592466 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" event={"ID":"86d99a3b-3579-431d-8bd8-958afee0a9a0","Type":"ContainerStarted","Data":"92977de93d76b3bc7a6bbb39babf498b91d0de8c3dc3d0ba77904d36bd5933bd"} Oct 30 13:06:11.592608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:11.592488 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:11.645303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:11.645221 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" podStartSLOduration=1.645206298 podStartE2EDuration="1.645206298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:11.644685649 +0000 UTC m=+1989.771486547" watchObservedRunningTime="2026-10-30 13:06:11.645206298 +0000 UTC m=+1989.772007196" Oct 30 13:06:12.809310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:12.809243 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:14.198036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.197968 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:06:14.344593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.344556 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-kube-api-access-8g4kd\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-kube-api-access-8g4kd\") pod \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " Oct 30 13:06:14.344804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.344776 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-token\") pod \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.345007 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-envoy\") pod \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.345074 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-podinfo\") pod \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.345149 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-data\") pod \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.345193 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-socket\") pod \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.345419 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-certs\") pod \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.345449 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-credential-socket\") pod \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.345486 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istiod-ca-cert\") pod \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\" (UID: \"f9f3746b-ffc4-4523-94a4-4c68b3eaa397\") " Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.345676 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-data" pod "f9f3746b-ffc4-4523-94a4-4c68b3eaa397" (UID: "f9f3746b-ffc4-4523-94a4-4c68b3eaa397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.346006 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istiod-ca-cert" pod "f9f3746b-ffc4-4523-94a4-4c68b3eaa397" (UID: "f9f3746b-ffc4-4523-94a4-4c68b3eaa397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.346237 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-socket" pod "f9f3746b-ffc4-4523-94a4-4c68b3eaa397" (UID: "f9f3746b-ffc4-4523-94a4-4c68b3eaa397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.346366 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.346393 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.346501 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-certs" pod "f9f3746b-ffc4-4523-94a4-4c68b3eaa397" (UID: "f9f3746b-ffc4-4523-94a4-4c68b3eaa397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:14.350312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.346635 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-credential-socket" pod "f9f3746b-ffc4-4523-94a4-4c68b3eaa397" (UID: "f9f3746b-ffc4-4523-94a4-4c68b3eaa397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:14.367642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.361832 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-podinfo" pod "f9f3746b-ffc4-4523-94a4-4c68b3eaa397" (UID: "f9f3746b-ffc4-4523-94a4-4c68b3eaa397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:06:14.387699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.387636 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-token" pod "f9f3746b-ffc4-4523-94a4-4c68b3eaa397" (UID: "f9f3746b-ffc4-4523-94a4-4c68b3eaa397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:14.392295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.389503 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-envoy" pod "f9f3746b-ffc4-4523-94a4-4c68b3eaa397" (UID: "f9f3746b-ffc4-4523-94a4-4c68b3eaa397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:14.393302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.393243 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-kube-api-access-8g4kd" pod "f9f3746b-ffc4-4523-94a4-4c68b3eaa397" (UID: "f9f3746b-ffc4-4523-94a4-4c68b3eaa397"). InnerVolumeSpecName "kube-api-access-8g4kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:14.410186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.410153 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:14.447176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.447138 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:14.447176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.447170 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:14.447176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.447180 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:14.447441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.447190 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g4kd\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-kube-api-access-8g4kd\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:14.447441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.447199 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:14.447441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.447207 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:14.447441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.447214 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9f3746b-ffc4-4523-94a4-4c68b3eaa397-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:14.613208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.613126 5515 generic.go:356] "Generic (PLEG): container finished" podID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerID="c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30" exitCode=0 Oct 30 13:06:14.613208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.613182 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" Oct 30 13:06:14.613406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.613217 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" event={"ID":"f9f3746b-ffc4-4523-94a4-4c68b3eaa397","Type":"ContainerDied","Data":"c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30"} Oct 30 13:06:14.613406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.613296 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t" event={"ID":"f9f3746b-ffc4-4523-94a4-4c68b3eaa397","Type":"ContainerDied","Data":"3250e73304e9e9ebed83c568356352811ca4e65040f10e6cdd256e7fc72f1295"} Oct 30 13:06:14.613406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.613320 5515 scope.go:122] "RemoveContainer" containerID="c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30" Oct 30 13:06:14.624029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.624012 5515 scope.go:122] "RemoveContainer" containerID="cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc" Oct 30 13:06:14.633370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.633313 5515 scope.go:122] "RemoveContainer" containerID="c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30" Oct 30 13:06:14.633620 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:14.633601 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30\": container with ID starting with c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30 not found: ID does not exist" containerID="c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30" Oct 30 13:06:14.633665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.633631 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30"} err="failed to get container status \"c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30\": rpc error: code = NotFound desc = could not find container \"c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30\": container with ID starting with c4549f73ff1c5ac719c8c80ed251f8ad706ba345c3b61f8ea27247f50e279a30 not found: ID does not exist" Oct 30 13:06:14.633665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.633650 5515 scope.go:122] "RemoveContainer" containerID="cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc" Oct 30 13:06:14.633885 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:14.633868 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc\": container with ID starting with cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc not found: ID does not exist" containerID="cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc" Oct 30 13:06:14.633927 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.633895 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc"} err="failed to get container status \"cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc\": rpc error: code = NotFound desc = could not find container \"cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc\": container with ID starting with cc1e935f5573d8957e154f7b0961a718ff928f65c3b8c17827a5fe8455ee46dc not found: ID does not exist" Oct 30 13:06:14.671173 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.671135 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t"] Oct 30 13:06:14.680143 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:14.680116 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f0f50-openshift-default-688859fb8f-k4w4t"] Oct 30 13:06:15.812068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:15.812010 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm"] Oct 30 13:06:15.812540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:15.812286 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" podUID="25749efb-4232-406d-aae6-6e6a802b2333" containerName="payload-processing" containerID="cri-o://f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c" gracePeriod=30 Oct 30 13:06:15.827547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:15.827514 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22"] Oct 30 13:06:15.827969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:15.827933 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" podUID="60554da6-b176-4d64-a077-0dc191618029" containerName="payload-pre-processing" containerID="cri-o://dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3" gracePeriod=30 Oct 30 13:06:15.863025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:15.862987 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.014216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.013811 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" path="/var/lib/kubelet/pods/f9f3746b-ffc4-4523-94a4-4c68b3eaa397/volumes" Oct 30 13:06:16.120401 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.116708 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.289873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.289848 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:06:16.292736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.292710 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:06:16.385410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.385382 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25749efb-4232-406d-aae6-6e6a802b2333-kube-api-access-lwrnm\" (UniqueName: \"kubernetes.io/projected/25749efb-4232-406d-aae6-6e6a802b2333-kube-api-access-lwrnm\") pod \"25749efb-4232-406d-aae6-6e6a802b2333\" (UID: \"25749efb-4232-406d-aae6-6e6a802b2333\") " Oct 30 13:06:16.385587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.385436 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60554da6-b176-4d64-a077-0dc191618029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60554da6-b176-4d64-a077-0dc191618029-plugins-config-volume\") pod \"60554da6-b176-4d64-a077-0dc191618029\" (UID: \"60554da6-b176-4d64-a077-0dc191618029\") " Oct 30 13:06:16.385587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.385487 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60554da6-b176-4d64-a077-0dc191618029-kube-api-access-458rw\" (UniqueName: \"kubernetes.io/projected/60554da6-b176-4d64-a077-0dc191618029-kube-api-access-458rw\") pod \"60554da6-b176-4d64-a077-0dc191618029\" (UID: \"60554da6-b176-4d64-a077-0dc191618029\") " Oct 30 13:06:16.385698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.385669 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25749efb-4232-406d-aae6-6e6a802b2333-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25749efb-4232-406d-aae6-6e6a802b2333-plugins-config-volume\") pod \"25749efb-4232-406d-aae6-6e6a802b2333\" (UID: \"25749efb-4232-406d-aae6-6e6a802b2333\") " Oct 30 13:06:16.385842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.385803 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60554da6-b176-4d64-a077-0dc191618029-plugins-config-volume" pod "60554da6-b176-4d64-a077-0dc191618029" (UID: "60554da6-b176-4d64-a077-0dc191618029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:16.386052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.386029 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25749efb-4232-406d-aae6-6e6a802b2333-plugins-config-volume" pod "25749efb-4232-406d-aae6-6e6a802b2333" (UID: "25749efb-4232-406d-aae6-6e6a802b2333"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:16.386549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.386531 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60554da6-b176-4d64-a077-0dc191618029-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:16.386659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.386551 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25749efb-4232-406d-aae6-6e6a802b2333-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:16.387898 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.387843 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25749efb-4232-406d-aae6-6e6a802b2333-kube-api-access-lwrnm" pod "25749efb-4232-406d-aae6-6e6a802b2333" (UID: "25749efb-4232-406d-aae6-6e6a802b2333"). InnerVolumeSpecName "kube-api-access-lwrnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:16.392012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.391988 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60554da6-b176-4d64-a077-0dc191618029-kube-api-access-458rw" pod "60554da6-b176-4d64-a077-0dc191618029" (UID: "60554da6-b176-4d64-a077-0dc191618029"). InnerVolumeSpecName "kube-api-access-458rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:16.442244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.442157 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.488097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.488055 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwrnm\" (UniqueName: \"kubernetes.io/projected/25749efb-4232-406d-aae6-6e6a802b2333-kube-api-access-lwrnm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:16.488097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.488091 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458rw\" (UniqueName: \"kubernetes.io/projected/60554da6-b176-4d64-a077-0dc191618029-kube-api-access-458rw\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:16.628430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.628394 5515 generic.go:356] "Generic (PLEG): container finished" podID="25749efb-4232-406d-aae6-6e6a802b2333" containerID="f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c" exitCode=0 Oct 30 13:06:16.628638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.628442 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" Oct 30 13:06:16.628638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.628481 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" event={"ID":"25749efb-4232-406d-aae6-6e6a802b2333","Type":"ContainerDied","Data":"f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c"} Oct 30 13:06:16.628638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.628538 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm" event={"ID":"25749efb-4232-406d-aae6-6e6a802b2333","Type":"ContainerDied","Data":"ec60b63901a35c81d3c06597d269fd5cfcaef7d47209da3d3a288877817121a0"} Oct 30 13:06:16.628638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.628566 5515 scope.go:122] "RemoveContainer" containerID="f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c" Oct 30 13:06:16.630082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.629948 5515 generic.go:356] "Generic (PLEG): container finished" podID="60554da6-b176-4d64-a077-0dc191618029" containerID="dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3" exitCode=0 Oct 30 13:06:16.630082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.629981 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" event={"ID":"60554da6-b176-4d64-a077-0dc191618029","Type":"ContainerDied","Data":"dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3"} Oct 30 13:06:16.630082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.630002 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" Oct 30 13:06:16.630082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.630035 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22" event={"ID":"60554da6-b176-4d64-a077-0dc191618029","Type":"ContainerDied","Data":"f3c08bc88808f40ab13088c736d7626225c3f355e943af95de46ae0b94f260db"} Oct 30 13:06:16.641935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.641907 5515 scope.go:122] "RemoveContainer" containerID="f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c" Oct 30 13:06:16.642459 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:16.642425 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c\": container with ID starting with f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c not found: ID does not exist" containerID="f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c" Oct 30 13:06:16.642579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.642468 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c"} err="failed to get container status \"f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c\": rpc error: code = NotFound desc = could not find container \"f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c\": container with ID starting with f4cc032ae59eeb0a55ce1a24d0578493915ad41af7547925d217153f2383624c not found: ID does not exist" Oct 30 13:06:16.642579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.642493 5515 scope.go:122] "RemoveContainer" containerID="dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3" Oct 30 13:06:16.654170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.654149 5515 scope.go:122] "RemoveContainer" containerID="dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3" Oct 30 13:06:16.654603 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:16.654582 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3\": container with ID starting with dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3 not found: ID does not exist" containerID="dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3" Oct 30 13:06:16.654656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.654614 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3"} err="failed to get container status \"dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3\": rpc error: code = NotFound desc = could not find container \"dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3\": container with ID starting with dac1bc08e4867d33cbacddf12b2f95cd462377e6f855a8002ebb99539d0b08e3 not found: ID does not exist" Oct 30 13:06:16.660651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.660622 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.682919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.682887 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm"] Oct 30 13:06:16.694377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.694345 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-405eb1-7869d65754-z8bgm"] Oct 30 13:06:16.730397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.730362 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22"] Oct 30 13:06:16.733436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:16.733412 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-c5d464c8c-76r22"] Oct 30 13:06:17.220663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:17.220631 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:17.981478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:17.981447 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq"] Oct 30 13:06:17.981700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:17.981645 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" podUID="d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" containerName="maas-api" containerID="cri-o://1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52" gracePeriod=30 Oct 30 13:06:18.013660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.013625 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25749efb-4232-406d-aae6-6e6a802b2333" path="/var/lib/kubelet/pods/25749efb-4232-406d-aae6-6e6a802b2333/volumes" Oct 30 13:06:18.014240 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.014217 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60554da6-b176-4d64-a077-0dc191618029" path="/var/lib/kubelet/pods/60554da6-b176-4d64-a077-0dc191618029/volumes" Oct 30 13:06:18.152385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.152347 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:18.381341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.378844 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:06:18.545347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.541823 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-kube-api-access-mlb6f\" (UniqueName: \"kubernetes.io/projected/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-kube-api-access-mlb6f\") pod \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " Oct 30 13:06:18.545347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.541889 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-openshift-service-ca\") pod \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " Oct 30 13:06:18.545347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.541934 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-metrics-certs\") pod \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " Oct 30 13:06:18.545347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.542042 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls\") pod \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\" (UID: \"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a\") " Oct 30 13:06:18.545347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.544017 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-openshift-service-ca" pod "d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" (UID: "d16d4d48-0912-4b0a-ac44-7d3e6b2b298a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:18.546559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.546523 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-metrics-certs" pod "d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" (UID: "d16d4d48-0912-4b0a-ac44-7d3e6b2b298a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:18.547679 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.547651 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls" pod "d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" (UID: "d16d4d48-0912-4b0a-ac44-7d3e6b2b298a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:18.548243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.548216 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-kube-api-access-mlb6f" pod "d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" (UID: "d16d4d48-0912-4b0a-ac44-7d3e6b2b298a"). InnerVolumeSpecName "kube-api-access-mlb6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:18.574082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.574047 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:06:18.574300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.574241 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:06:18.643701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.643633 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlb6f\" (UniqueName: \"kubernetes.io/projected/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-kube-api-access-mlb6f\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:18.643701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.643663 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:18.643701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.643678 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:18.643701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.643690 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:18.647667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.647626 5515 generic.go:356] "Generic (PLEG): container finished" podID="d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" containerID="1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52" exitCode=0 Oct 30 13:06:18.647875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.647822 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" event={"ID":"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a","Type":"ContainerDied","Data":"1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52"} Oct 30 13:06:18.647939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.647887 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" event={"ID":"d16d4d48-0912-4b0a-ac44-7d3e6b2b298a","Type":"ContainerDied","Data":"5a3c37c79956fcd8af74fe4bc527f40af43abd315fa886df1a564482773b2f9e"} Oct 30 13:06:18.647939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.647922 5515 scope.go:122] "RemoveContainer" containerID="1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52" Oct 30 13:06:18.648052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.648025 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq" Oct 30 13:06:18.659648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.659622 5515 scope.go:122] "RemoveContainer" containerID="1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52" Oct 30 13:06:18.662756 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:18.662716 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52\": container with ID starting with 1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52 not found: ID does not exist" containerID="1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52" Oct 30 13:06:18.662889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.662766 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52"} err="failed to get container status \"1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52\": rpc error: code = NotFound desc = could not find container \"1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52\": container with ID starting with 1cd8b01b72de2ea72d356a41195d8605cf1acc7a4654e59a3f454602e1e28c52 not found: ID does not exist" Oct 30 13:06:18.699724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.699682 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq"] Oct 30 13:06:18.705743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.705708 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a236d-6887d9997c-5x8wq"] Oct 30 13:06:18.747908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.747874 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:18.990492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:18.990461 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:19.399643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:19.399608 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:20.001824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:20.001786 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" path="/var/lib/kubelet/pods/d16d4d48-0912-4b0a-ac44-7d3e6b2b298a/volumes" Oct 30 13:06:20.609574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:20.609538 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:21.721812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:21.721768 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk"] Oct 30 13:06:21.722393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:21.722010 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" podUID="19bf50b1-2822-4430-8305-a159a5a22912" containerName="maas-api" containerID="cri-o://fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846" gracePeriod=30 Oct 30 13:06:21.893922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:21.893891 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:22.015701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.015673 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:06:22.120426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.120386 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-metrics-certs\") pod \"19bf50b1-2822-4430-8305-a159a5a22912\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " Oct 30 13:06:22.120613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.120565 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-maas-api-tls\") pod \"19bf50b1-2822-4430-8305-a159a5a22912\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " Oct 30 13:06:22.120678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.120654 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bf50b1-2822-4430-8305-a159a5a22912-kube-api-access-dfr6s\" (UniqueName: \"kubernetes.io/projected/19bf50b1-2822-4430-8305-a159a5a22912-kube-api-access-dfr6s\") pod \"19bf50b1-2822-4430-8305-a159a5a22912\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " Oct 30 13:06:22.120744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.120709 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19bf50b1-2822-4430-8305-a159a5a22912-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bf50b1-2822-4430-8305-a159a5a22912-openshift-service-ca\") pod \"19bf50b1-2822-4430-8305-a159a5a22912\" (UID: \"19bf50b1-2822-4430-8305-a159a5a22912\") " Oct 30 13:06:22.121114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.121080 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bf50b1-2822-4430-8305-a159a5a22912-openshift-service-ca" pod "19bf50b1-2822-4430-8305-a159a5a22912" (UID: "19bf50b1-2822-4430-8305-a159a5a22912"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:22.121324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.121306 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19bf50b1-2822-4430-8305-a159a5a22912-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.122789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.122766 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-metrics-certs" pod "19bf50b1-2822-4430-8305-a159a5a22912" (UID: "19bf50b1-2822-4430-8305-a159a5a22912"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:22.123211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.123182 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-maas-api-tls" pod "19bf50b1-2822-4430-8305-a159a5a22912" (UID: "19bf50b1-2822-4430-8305-a159a5a22912"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:22.123211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.123187 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bf50b1-2822-4430-8305-a159a5a22912-kube-api-access-dfr6s" pod "19bf50b1-2822-4430-8305-a159a5a22912" (UID: "19bf50b1-2822-4430-8305-a159a5a22912"). InnerVolumeSpecName "kube-api-access-dfr6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:22.151904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.151876 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:22.233346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.231297 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfr6s\" (UniqueName: \"kubernetes.io/projected/19bf50b1-2822-4430-8305-a159a5a22912-kube-api-access-dfr6s\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.233346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.231340 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.233346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.231353 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19bf50b1-2822-4430-8305-a159a5a22912-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.411299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.411223 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:22.608345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.606091 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:06:22.676735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.676697 5515 generic.go:356] "Generic (PLEG): container finished" podID="19bf50b1-2822-4430-8305-a159a5a22912" containerID="fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846" exitCode=0 Oct 30 13:06:22.676985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.676793 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" Oct 30 13:06:22.676985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.676882 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" event={"ID":"19bf50b1-2822-4430-8305-a159a5a22912","Type":"ContainerDied","Data":"fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846"} Oct 30 13:06:22.676985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.676934 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk" event={"ID":"19bf50b1-2822-4430-8305-a159a5a22912","Type":"ContainerDied","Data":"20a721fb7af7f8612c1c2232bf60d602717a65253c5abec066701dc0a5065a45"} Oct 30 13:06:22.676985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.676964 5515 scope.go:122] "RemoveContainer" containerID="fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846" Oct 30 13:06:22.698356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.698325 5515 scope.go:122] "RemoveContainer" containerID="fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846" Oct 30 13:06:22.698736 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:22.698715 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846\": container with ID starting with fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846 not found: ID does not exist" containerID="fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846" Oct 30 13:06:22.698825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.698747 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846"} err="failed to get container status \"fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846\": rpc error: code = NotFound desc = could not find container \"fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846\": container with ID starting with fb9556b65842659fd9bb02ab3f9a85f8987e9175cc7a72a7b02d3b5938c31846 not found: ID does not exist" Oct 30 13:06:22.734112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.734074 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk"] Oct 30 13:06:22.739533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.739503 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1ee32-8c8d7fc79-cdhrk"] Oct 30 13:06:22.744103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.744073 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:22.962417 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:22.962383 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:23.212525 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.212440 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn"] Oct 30 13:06:23.213082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213066 5515 cpu_manager.go:414] "containerMap: removing container" podUID="60554da6-b176-4d64-a077-0dc191618029" containerName="payload-pre-processing" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213083 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60554da6-b176-4d64-a077-0dc191618029" containerName="payload-pre-processing" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213101 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" containerName="maas-api" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213106 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" containerName="maas-api" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213115 5515 cpu_manager.go:414] "containerMap: removing container" podUID="25749efb-4232-406d-aae6-6e6a802b2333" containerName="payload-processing" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213120 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25749efb-4232-406d-aae6-6e6a802b2333" containerName="payload-processing" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213134 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213139 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213146 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213151 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" Oct 30 13:06:23.213156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213159 5515 cpu_manager.go:414] "containerMap: removing container" podUID="19bf50b1-2822-4430-8305-a159a5a22912" containerName="maas-api" Oct 30 13:06:23.213692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213164 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bf50b1-2822-4430-8305-a159a5a22912" containerName="maas-api" Oct 30 13:06:23.213692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213295 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="60554da6-b176-4d64-a077-0dc191618029" containerName="payload-pre-processing" Oct 30 13:06:23.213692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213306 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16d4d48-0912-4b0a-ac44-7d3e6b2b298a" containerName="maas-api" Oct 30 13:06:23.213692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213315 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="25749efb-4232-406d-aae6-6e6a802b2333" containerName="payload-processing" Oct 30 13:06:23.213692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213322 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" Oct 30 13:06:23.213692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213328 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f3746b-ffc4-4523-94a4-4c68b3eaa397" containerName="istio-proxy" Oct 30 13:06:23.213692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.213333 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bf50b1-2822-4430-8305-a159a5a22912" containerName="maas-api" Oct 30 13:06:23.218681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.218651 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.241993 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.241954 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn"] Oct 30 13:06:23.242187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.242160 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:23.244657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.244626 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-socket\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.244784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.244688 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shj7s\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-kube-api-access-shj7s\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.244841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.244813 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-data\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.244888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.244874 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-podinfo\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.245012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.244988 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f4bbae-0258-42ed-9d5f-95706d37e103-istiod-ca-cert\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.245063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.245035 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-envoy\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.245151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.245133 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-certs\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.245340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.245321 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-credential-socket\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.245402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.245369 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-token\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.247850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.247827 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6e0e74-openshift-default-dockercfg-zg9r2\"" Oct 30 13:06:23.346623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.346589 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shj7s\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-kube-api-access-shj7s\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.346844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.346645 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-data\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.346844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.346680 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-podinfo\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.346844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.346715 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f4bbae-0258-42ed-9d5f-95706d37e103-istiod-ca-cert\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.346844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.346745 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-envoy\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.346844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.346780 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-certs\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.347117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.346905 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-credential-socket\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.347117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.346944 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-token\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.347117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.347012 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-socket\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.347368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.347174 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-data\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.347368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.347304 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-certs\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.347469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.347383 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-socket\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.347558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.347539 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-credential-socket\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.347643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.347620 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f4bbae-0258-42ed-9d5f-95706d37e103-istiod-ca-cert\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.349164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.349147 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-envoy\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.349413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.349394 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-podinfo\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.373507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.373471 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shj7s\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-kube-api-access-shj7s\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.373507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.373496 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-token\") pod \"e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.501459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.499411 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:23.544140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.542839 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:23.723942 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.723909 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:23.893573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.893539 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn"] Oct 30 13:06:23.893929 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:06:23.893908 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f4bbae_0258_42ed_9d5f_95706d37e103.slice/crio-90289b357f4002d4ad2eb4e1d86d7a963e372d403d65fc6768e23dc3c4855d11 WatchSource:0}: Error finding container 90289b357f4002d4ad2eb4e1d86d7a963e372d403d65fc6768e23dc3c4855d11: Status 404 returned error can't find the container with id 90289b357f4002d4ad2eb4e1d86d7a963e372d403d65fc6768e23dc3c4855d11 Oct 30 13:06:23.896513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.896472 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:23.896627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.896543 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:23.896627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:23.896574 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:24.005039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:24.004992 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bf50b1-2822-4430-8305-a159a5a22912" path="/var/lib/kubelet/pods/19bf50b1-2822-4430-8305-a159a5a22912/volumes" Oct 30 13:06:24.693386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:24.693349 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" event={"ID":"90f4bbae-0258-42ed-9d5f-95706d37e103","Type":"ContainerStarted","Data":"dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62"} Oct 30 13:06:24.693571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:24.693411 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" event={"ID":"90f4bbae-0258-42ed-9d5f-95706d37e103","Type":"ContainerStarted","Data":"90289b357f4002d4ad2eb4e1d86d7a963e372d403d65fc6768e23dc3c4855d11"} Oct 30 13:06:24.802526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:24.802467 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:25.096602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:25.096511 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:25.499940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:25.499891 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:25.545714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:25.545678 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:25.624137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:25.624111 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:25.685655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:25.685602 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" podStartSLOduration=2.685587256 podStartE2EDuration="2.685587256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:24.775865937 +0000 UTC m=+2002.902666836" watchObservedRunningTime="2026-10-30 13:06:25.685587256 +0000 UTC m=+2003.812388153" Oct 30 13:06:25.700344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:25.700313 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:25.701388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:25.701370 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:06:25.929583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:25.929547 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:26.231416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.231335 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx"] Oct 30 13:06:26.235456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.235434 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.261666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.261638 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-76c107-openshift-default-dockercfg-d4wwv\"" Oct 30 13:06:26.279756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.279721 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx"] Oct 30 13:06:26.318531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.318495 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b15247e-2417-4458-aada-9598ad342b4e-istio-podinfo\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.318712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.318551 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-istio-token\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.318712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.318644 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b15247e-2417-4458-aada-9598ad342b4e-istiod-ca-cert\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.318712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.318677 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rqb\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-kube-api-access-n4rqb\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.318833 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.318754 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-data\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.318833 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.318793 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-certs\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.318914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.318901 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-credential-socket\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.319020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.319009 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-socket\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.319056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.319032 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-envoy\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.410347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.410313 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:26.422258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422204 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-socket\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.422452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422302 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-envoy\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.422452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422347 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b15247e-2417-4458-aada-9598ad342b4e-istio-podinfo\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.422452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422392 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-istio-token\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.422452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422445 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b15247e-2417-4458-aada-9598ad342b4e-istiod-ca-cert\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.422689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422491 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rqb\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-kube-api-access-n4rqb\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.422689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422536 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-data\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.422689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422583 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-certs\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.422689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422681 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-credential-socket\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.422896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.422709 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-socket\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.432765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.432130 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-envoy\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.432765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.432712 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b15247e-2417-4458-aada-9598ad342b4e-istiod-ca-cert\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.433378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.433329 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-certs\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.433728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.433690 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-credential-socket\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.433996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.433977 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-data\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.445440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.444982 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b15247e-2417-4458-aada-9598ad342b4e-istio-podinfo\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.464298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.462826 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rqb\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-kube-api-access-n4rqb\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.491344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.482546 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-istio-token\") pod \"e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.556987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.555556 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:26.648959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.647786 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:26.915666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.915630 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:26.961690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.954826 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx"] Oct 30 13:06:26.986263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.986211 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:26.986584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.986561 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:26.986755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:26.986737 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:27.153650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.153615 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:27.394096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.394060 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb"] Oct 30 13:06:27.399385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.399354 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.421354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.421320 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4a97e9-openshift-default-dockercfg-bj964\"" Oct 30 13:06:27.426007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.425970 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb"] Oct 30 13:06:27.471080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.471038 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-envoy\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.471407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.471148 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-credential-socket\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.471407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.471335 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-istio-token\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.471407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.471394 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117c8a18-3210-4f68-9918-0e92612dadaa-istio-podinfo\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.471626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.471483 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117c8a18-3210-4f68-9918-0e92612dadaa-istiod-ca-cert\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.471626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.471519 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-data\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.471626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.471557 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrch4\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-kube-api-access-jrch4\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.471861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.471667 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-socket\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.471861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.471742 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-certs\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.492869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.492809 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:27.598295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.597168 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-socket\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.597228 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-certs\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.597334 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-envoy\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.597425 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-credential-socket\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.597522 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-istio-token\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.597569 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117c8a18-3210-4f68-9918-0e92612dadaa-istio-podinfo\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.597630 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117c8a18-3210-4f68-9918-0e92612dadaa-istiod-ca-cert\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.597670 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-data\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.597718 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrch4\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-kube-api-access-jrch4\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.598625 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-socket\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.598971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.598951 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-certs\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.611293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.600076 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117c8a18-3210-4f68-9918-0e92612dadaa-istiod-ca-cert\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.611293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.600367 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-credential-socket\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.611293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.600642 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-data\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.611293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.603448 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-envoy\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.634348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.626955 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117c8a18-3210-4f68-9918-0e92612dadaa-istio-podinfo\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.634602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.629489 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-istio-token\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.634747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.633680 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrch4\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-kube-api-access-jrch4\") pod \"e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.718660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.718571 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:27.728037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.728005 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:27.737388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.737348 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" event={"ID":"5b15247e-2417-4458-aada-9598ad342b4e","Type":"ContainerStarted","Data":"2c64d78b327475d74da78e143a2d6078b03e5a1ecfce682b528e6c4c413bd4e4"} Oct 30 13:06:27.737540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.737405 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" event={"ID":"5b15247e-2417-4458-aada-9598ad342b4e","Type":"ContainerStarted","Data":"94fc4e2b4a880872fde59e82d3737ed9faa498fdfa92bbda27883b208285cc9b"} Oct 30 13:06:27.789367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:27.789310 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" podStartSLOduration=1.789293019 podStartE2EDuration="1.789293019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:27.786656485 +0000 UTC m=+2005.913457384" watchObservedRunningTime="2026-10-30 13:06:27.789293019 +0000 UTC m=+2005.916093916" Oct 30 13:06:28.027040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.026904 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:28.061970 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:06:28.061935 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117c8a18_3210_4f68_9918_0e92612dadaa.slice/crio-8ccf2a373c3f3e33c9206e28294b4287919196c0db74be7aa61bbc32280d3841 WatchSource:0}: Error finding container 8ccf2a373c3f3e33c9206e28294b4287919196c0db74be7aa61bbc32280d3841: Status 404 returned error can't find the container with id 8ccf2a373c3f3e33c9206e28294b4287919196c0db74be7aa61bbc32280d3841 Oct 30 13:06:28.061970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.061948 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb"] Oct 30 13:06:28.064668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.064629 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:28.064761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.064701 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:28.064761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.064734 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:06:28.269292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.267779 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:28.556745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.556717 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:28.562505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.562478 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:28.745744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.745697 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" event={"ID":"117c8a18-3210-4f68-9918-0e92612dadaa","Type":"ContainerStarted","Data":"402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c"} Oct 30 13:06:28.745956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.745774 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" event={"ID":"117c8a18-3210-4f68-9918-0e92612dadaa","Type":"ContainerStarted","Data":"8ccf2a373c3f3e33c9206e28294b4287919196c0db74be7aa61bbc32280d3841"} Oct 30 13:06:28.746036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.746005 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:28.747409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.747387 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:06:28.793786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:28.793724 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podStartSLOduration=1.793710317 podStartE2EDuration="1.793710317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:28.793176269 +0000 UTC m=+2006.919977167" watchObservedRunningTime="2026-10-30 13:06:28.793710317 +0000 UTC m=+2006.920511215" Oct 30 13:06:29.059365 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:29.059331 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:06:29.288469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.288437 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:29.430129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.430093 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45"] Oct 30 13:06:29.434584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.434555 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:29.490292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.476443 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjmm\" (UniqueName: \"kubernetes.io/projected/e9909d73-5c03-435c-8700-f709e8186f7a-kube-api-access-wvjmm\") pod \"payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45\" (UID: \"e9909d73-5c03-435c-8700-f709e8186f7a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:29.490292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.476570 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9909d73-5c03-435c-8700-f709e8186f7a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45\" (UID: \"e9909d73-5c03-435c-8700-f709e8186f7a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:29.514124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.505745 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6e0e74-dockercfg-lvwgn\"" Oct 30 13:06:29.514124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.505878 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6e0e74\"" Oct 30 13:06:29.552603 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.552561 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45"] Oct 30 13:06:29.579295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.578777 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:29.585348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.580337 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjmm\" (UniqueName: \"kubernetes.io/projected/e9909d73-5c03-435c-8700-f709e8186f7a-kube-api-access-wvjmm\") pod \"payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45\" (UID: \"e9909d73-5c03-435c-8700-f709e8186f7a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:29.585348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.580460 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9909d73-5c03-435c-8700-f709e8186f7a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45\" (UID: \"e9909d73-5c03-435c-8700-f709e8186f7a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:29.585348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.581623 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9909d73-5c03-435c-8700-f709e8186f7a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45\" (UID: \"e9909d73-5c03-435c-8700-f709e8186f7a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:29.621086 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.621032 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjmm\" (UniqueName: \"kubernetes.io/projected/e9909d73-5c03-435c-8700-f709e8186f7a-kube-api-access-wvjmm\") pod \"payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45\" (UID: \"e9909d73-5c03-435c-8700-f709e8186f7a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:29.719814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.719723 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:29.725567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.725535 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:29.752430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.752399 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:29.753537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.753518 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:06:29.902445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:29.902411 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:30.225056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:30.224982 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45"] Oct 30 13:06:30.225158 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:06:30.225046 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9909d73_5c03_435c_8700_f709e8186f7a.slice/crio-5b9e746c0ede92d89d00b3e04b297a39bd50c6bde68bd721d487cb99dcbe2af5 WatchSource:0}: Error finding container 5b9e746c0ede92d89d00b3e04b297a39bd50c6bde68bd721d487cb99dcbe2af5: Status 404 returned error can't find the container with id 5b9e746c0ede92d89d00b3e04b297a39bd50c6bde68bd721d487cb99dcbe2af5 Oct 30 13:06:30.578590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:30.578499 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:30.780644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:30.779296 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" event={"ID":"e9909d73-5c03-435c-8700-f709e8186f7a","Type":"ContainerStarted","Data":"84944be9206419cf73c6ad2d2b0bbe7093e02cf953f58a14fc73599e458073c0"} Oct 30 13:06:30.780644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:30.779387 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" event={"ID":"e9909d73-5c03-435c-8700-f709e8186f7a","Type":"ContainerStarted","Data":"5b9e746c0ede92d89d00b3e04b297a39bd50c6bde68bd721d487cb99dcbe2af5"} Oct 30 13:06:30.780644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:30.780003 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:30.795394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:30.793775 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:30.821587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:30.821533 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" podStartSLOduration=1.821519455 podStartE2EDuration="1.821519455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:30.819784496 +0000 UTC m=+2008.946585411" watchObservedRunningTime="2026-10-30 13:06:30.821519455 +0000 UTC m=+2008.948320380" Oct 30 13:06:31.175870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:31.175834 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:31.454470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:31.454387 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:31.701841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:31.701753 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:32.465168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:32.465133 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:32.746179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:32.745718 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:32.973217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:32.973176 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:33.222630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:33.222593 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:33.469580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:33.469536 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:33.701777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:33.701742 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:33.981054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:33.980965 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.304409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.304322 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.385233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.385196 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx"] Oct 30 13:06:34.389650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.389624 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:34.407174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.407147 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4a97e9\"" Oct 30 13:06:34.419439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.419408 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4a97e9-dockercfg-4gtrj\"" Oct 30 13:06:34.424545 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.424513 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx"] Oct 30 13:06:34.440623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.440582 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx\" (UID: \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:34.440804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.440702 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgswx\" (UniqueName: \"kubernetes.io/projected/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-kube-api-access-rgswx\") pod \"payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx\" (UID: \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:34.502576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.502539 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c"] Oct 30 13:06:34.546358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.543502 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgswx\" (UniqueName: \"kubernetes.io/projected/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-kube-api-access-rgswx\") pod \"payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx\" (UID: \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:34.546358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.543769 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx\" (UID: \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:34.546358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.545301 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx\" (UID: \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:34.609799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.606294 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.617733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.612918 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgswx\" (UniqueName: \"kubernetes.io/projected/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-kube-api-access-rgswx\") pod \"payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx\" (UID: \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:34.619259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.618428 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:34.619695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.619670 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c"] Oct 30 13:06:34.711309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.705171 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:34.759730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.753594 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60c9615-42b8-47bd-9547-3abb6e39171d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c\" (UID: \"a60c9615-42b8-47bd-9547-3abb6e39171d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:34.759730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.753841 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxrf\" (UniqueName: \"kubernetes.io/projected/a60c9615-42b8-47bd-9547-3abb6e39171d-kube-api-access-zdxrf\") pod \"payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c\" (UID: \"a60c9615-42b8-47bd-9547-3abb6e39171d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:34.855745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.855039 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxrf\" (UniqueName: \"kubernetes.io/projected/a60c9615-42b8-47bd-9547-3abb6e39171d-kube-api-access-zdxrf\") pod \"payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c\" (UID: \"a60c9615-42b8-47bd-9547-3abb6e39171d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:34.855745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.855132 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60c9615-42b8-47bd-9547-3abb6e39171d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c\" (UID: \"a60c9615-42b8-47bd-9547-3abb6e39171d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:34.856891 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.856828 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60c9615-42b8-47bd-9547-3abb6e39171d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c\" (UID: \"a60c9615-42b8-47bd-9547-3abb6e39171d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:34.880959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.880882 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxrf\" (UniqueName: \"kubernetes.io/projected/a60c9615-42b8-47bd-9547-3abb6e39171d-kube-api-access-zdxrf\") pod \"payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c\" (UID: \"a60c9615-42b8-47bd-9547-3abb6e39171d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:34.982136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:34.982101 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:35.153436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.135931 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx"] Oct 30 13:06:35.153436 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:06:35.145076 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d34394d_ac4a_4bc1_9cbb_5b2c6d215587.slice/crio-c70062608e29ac0598226ce9ea2210262b8f992e13919ae8b683c1c31fecbdb3 WatchSource:0}: Error finding container c70062608e29ac0598226ce9ea2210262b8f992e13919ae8b683c1c31fecbdb3: Status 404 returned error can't find the container with id c70062608e29ac0598226ce9ea2210262b8f992e13919ae8b683c1c31fecbdb3 Oct 30 13:06:35.207847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.207817 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:35.285717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.285687 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c"] Oct 30 13:06:35.286201 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:06:35.286181 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60c9615_42b8_47bd_9547_3abb6e39171d.slice/crio-917d9117efbbde3fb9d3e17a56287584e8ca8b58e63b3f93b4092ba3ed63854e WatchSource:0}: Error finding container 917d9117efbbde3fb9d3e17a56287584e8ca8b58e63b3f93b4092ba3ed63854e: Status 404 returned error can't find the container with id 917d9117efbbde3fb9d3e17a56287584e8ca8b58e63b3f93b4092ba3ed63854e Oct 30 13:06:35.820636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.820588 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" event={"ID":"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587","Type":"ContainerStarted","Data":"249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2"} Oct 30 13:06:35.821146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.820650 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" event={"ID":"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587","Type":"ContainerStarted","Data":"c70062608e29ac0598226ce9ea2210262b8f992e13919ae8b683c1c31fecbdb3"} Oct 30 13:06:35.821146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.820682 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:35.822253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.822220 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" event={"ID":"a60c9615-42b8-47bd-9547-3abb6e39171d","Type":"ContainerStarted","Data":"d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3"} Oct 30 13:06:35.822426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.822286 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" event={"ID":"a60c9615-42b8-47bd-9547-3abb6e39171d","Type":"ContainerStarted","Data":"917d9117efbbde3fb9d3e17a56287584e8ca8b58e63b3f93b4092ba3ed63854e"} Oct 30 13:06:35.822426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.822346 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:35.864305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.864233 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" podStartSLOduration=1.8642201950000001 podStartE2EDuration="1.864220195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:35.863338338 +0000 UTC m=+2013.990139246" watchObservedRunningTime="2026-10-30 13:06:35.864220195 +0000 UTC m=+2013.991021092" Oct 30 13:06:35.904298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:35.904214 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" podStartSLOduration=1.9042007970000001 podStartE2EDuration="1.904200797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:35.903825376 +0000 UTC m=+2014.030626276" watchObservedRunningTime="2026-10-30 13:06:35.904200797 +0000 UTC m=+2014.031001700" Oct 30 13:06:36.909802 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:36.906427 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:37.127319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:37.127248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:37.360181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:37.360093 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:38.717107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:38.717071 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:38.965296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:38.961812 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:39.417882 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:39.417842 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:39.689968 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:39.689855 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:40.413104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:40.413049 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:42.382330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:42.381445 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:42.801005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:42.800927 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:06:43.847294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:43.843461 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:44.094150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:44.094102 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:45.544345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:45.544313 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:46.831741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:46.831709 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:06:46.832159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:46.831785 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:06:47.738699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:47.738655 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:48.031835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:48.031748 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:49.057590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:49.057554 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:06:49.191089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:49.191021 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:50.547020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.546974 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs"] Oct 30 13:06:50.553110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.553083 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.572935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.572902 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs"] Oct 30 13:06:50.574166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.574136 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4a97e9-serving-cert\"" Oct 30 13:06:50.590726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.590683 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce79343e-e824-44f4-b5ea-acb39dcdab89-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.590895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.590837 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-metrics-certs\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.591035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.591012 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.591196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.591170 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqkl\" (UniqueName: \"kubernetes.io/projected/ce79343e-e824-44f4-b5ea-acb39dcdab89-kube-api-access-rnqkl\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.693000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.692949 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.693158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.693058 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqkl\" (UniqueName: \"kubernetes.io/projected/ce79343e-e824-44f4-b5ea-acb39dcdab89-kube-api-access-rnqkl\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.693196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.693158 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce79343e-e824-44f4-b5ea-acb39dcdab89-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.693261 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.693205 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-metrics-certs\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.694031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.694002 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce79343e-e824-44f4-b5ea-acb39dcdab89-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.695718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.695693 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.695825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.695763 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-metrics-certs\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.717834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.717798 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqkl\" (UniqueName: \"kubernetes.io/projected/ce79343e-e824-44f4-b5ea-acb39dcdab89-kube-api-access-rnqkl\") pod \"maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:50.742620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.742586 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:50.870979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:50.870892 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:51.031279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:51.031226 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:51.198284 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:51.198240 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs"] Oct 30 13:06:51.199088 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:06:51.199060 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce79343e_e824_44f4_b5ea_acb39dcdab89.slice/crio-25a5577d896a16bad4f3f7d37161eec239fd99313f9caf88a42570088732f60a WatchSource:0}: Error finding container 25a5577d896a16bad4f3f7d37161eec239fd99313f9caf88a42570088732f60a: Status 404 returned error can't find the container with id 25a5577d896a16bad4f3f7d37161eec239fd99313f9caf88a42570088732f60a Oct 30 13:06:52.016688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:52.016654 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" event={"ID":"ce79343e-e824-44f4-b5ea-acb39dcdab89","Type":"ContainerStarted","Data":"53cf18f865c499d68a6ce5abd452202d44560f4ca4add20c5a73fa77b3fb1c5d"} Oct 30 13:06:52.017187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:52.016706 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" event={"ID":"ce79343e-e824-44f4-b5ea-acb39dcdab89","Type":"ContainerStarted","Data":"25a5577d896a16bad4f3f7d37161eec239fd99313f9caf88a42570088732f60a"} Oct 30 13:06:52.017187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:52.016726 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:52.063948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:52.063892 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" podStartSLOduration=1.7098103409999998 podStartE2EDuration="2.063876006s" totalImagesPullingTime="354.065665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:52.060004538 +0000 UTC m=+2030.186805441" watchObservedRunningTime="2026-10-30 13:06:52.063876006 +0000 UTC m=+2030.190676904" Oct 30 13:06:52.413958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:52.413904 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:54.135532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:54.135494 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:55.794228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:55.794191 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:57.062757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.062719 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj"] Oct 30 13:06:57.067129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.067110 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.090458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.090429 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8c35df-serving-cert\"" Oct 30 13:06:57.094464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.094427 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj"] Oct 30 13:06:57.195333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.195291 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-metrics-certs\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.195579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.195344 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.195579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.195432 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77400f5c-e08f-4bb2-adaa-831c954153b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.195691 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.195654 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mpv\" (UniqueName: \"kubernetes.io/projected/77400f5c-e08f-4bb2-adaa-831c954153b7-kube-api-access-k8mpv\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.276298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.273054 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:57.298293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.297472 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mpv\" (UniqueName: \"kubernetes.io/projected/77400f5c-e08f-4bb2-adaa-831c954153b7-kube-api-access-k8mpv\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.298293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.297535 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-metrics-certs\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.298293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.297576 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.298293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.297606 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77400f5c-e08f-4bb2-adaa-831c954153b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.298650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.298559 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77400f5c-e08f-4bb2-adaa-831c954153b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.304096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.304058 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-metrics-certs\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.308350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.304800 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.322373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.322302 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mpv\" (UniqueName: \"kubernetes.io/projected/77400f5c-e08f-4bb2-adaa-831c954153b7-kube-api-access-k8mpv\") pod \"maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.383459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.383426 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:57.472175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.472146 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:57.691145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:57.691118 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj"] Oct 30 13:06:57.691584 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:06:57.691553 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77400f5c_e08f_4bb2_adaa_831c954153b7.slice/crio-bf9a3a0b0d3877541db0d13462f22ba9a9c84b4fd89f66a531516e60382353b7 WatchSource:0}: Error finding container bf9a3a0b0d3877541db0d13462f22ba9a9c84b4fd89f66a531516e60382353b7: Status 404 returned error can't find the container with id bf9a3a0b0d3877541db0d13462f22ba9a9c84b4fd89f66a531516e60382353b7 Oct 30 13:06:58.116566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:58.083947 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" event={"ID":"77400f5c-e08f-4bb2-adaa-831c954153b7","Type":"ContainerStarted","Data":"bf9a3a0b0d3877541db0d13462f22ba9a9c84b4fd89f66a531516e60382353b7"} Oct 30 13:06:58.909684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:58.909648 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:58.990802 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:06:58.990767 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:06:59.032497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:59.032466 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:06:59.097062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:59.097008 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" event={"ID":"77400f5c-e08f-4bb2-adaa-831c954153b7","Type":"ContainerStarted","Data":"9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57"} Oct 30 13:06:59.097307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:59.097098 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:06:59.141315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:06:59.141240 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" podStartSLOduration=1.841029579 podStartE2EDuration="2.141226694s" totalImagesPullingTime="300.197115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:59.140454758 +0000 UTC m=+2037.267255656" watchObservedRunningTime="2026-10-30 13:06:59.141226694 +0000 UTC m=+2037.268027623" Oct 30 13:07:00.577872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:00.577824 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:02.694287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:02.694045 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:04.318293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:04.316816 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:05.163293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:05.145821 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:07:05.891296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:05.884890 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:08.077371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:08.077325 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45"] Oct 30 13:07:08.122892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:08.122849 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:08.357831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:08.357738 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:08.589064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:08.587857 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:09.951247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:09.951153 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:10.392650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:10.392601 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:11.268220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:11.268174 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:11.512489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:11.512458 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:11.740325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:11.740248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:11.743466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:11.743435 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:11.743578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:11.743515 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:13.078086 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.078017 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" podUID="e9909d73-5c03-435c-8700-f709e8186f7a" containerName="payload-processing" containerID="cri-o://84944be9206419cf73c6ad2d2b0bbe7093e02cf953f58a14fc73599e458073c0" gracePeriod=25 Oct 30 13:07:13.245733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.245696 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:13.359397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.359101 5515 generic.go:356] "Generic (PLEG): container finished" podID="e9909d73-5c03-435c-8700-f709e8186f7a" containerID="84944be9206419cf73c6ad2d2b0bbe7093e02cf953f58a14fc73599e458073c0" exitCode=0 Oct 30 13:07:13.359397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.359300 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" event={"ID":"e9909d73-5c03-435c-8700-f709e8186f7a","Type":"ContainerDied","Data":"84944be9206419cf73c6ad2d2b0bbe7093e02cf953f58a14fc73599e458073c0"} Oct 30 13:07:13.492561 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.492531 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:07:13.606765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.606677 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9909d73-5c03-435c-8700-f709e8186f7a-kube-api-access-wvjmm\" (UniqueName: \"kubernetes.io/projected/e9909d73-5c03-435c-8700-f709e8186f7a-kube-api-access-wvjmm\") pod \"e9909d73-5c03-435c-8700-f709e8186f7a\" (UID: \"e9909d73-5c03-435c-8700-f709e8186f7a\") " Oct 30 13:07:13.606765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.606726 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9909d73-5c03-435c-8700-f709e8186f7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9909d73-5c03-435c-8700-f709e8186f7a-plugins-config-volume\") pod \"e9909d73-5c03-435c-8700-f709e8186f7a\" (UID: \"e9909d73-5c03-435c-8700-f709e8186f7a\") " Oct 30 13:07:13.607163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.607137 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9909d73-5c03-435c-8700-f709e8186f7a-plugins-config-volume" pod "e9909d73-5c03-435c-8700-f709e8186f7a" (UID: "e9909d73-5c03-435c-8700-f709e8186f7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:13.607440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.607422 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9909d73-5c03-435c-8700-f709e8186f7a-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:07:13.609020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.608992 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9909d73-5c03-435c-8700-f709e8186f7a-kube-api-access-wvjmm" pod "e9909d73-5c03-435c-8700-f709e8186f7a" (UID: "e9909d73-5c03-435c-8700-f709e8186f7a"). InnerVolumeSpecName "kube-api-access-wvjmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:13.708829 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:13.708795 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvjmm\" (UniqueName: \"kubernetes.io/projected/e9909d73-5c03-435c-8700-f709e8186f7a-kube-api-access-wvjmm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:07:14.865325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.367611 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" event={"ID":"e9909d73-5c03-435c-8700-f709e8186f7a","Type":"ContainerDied","Data":"5b9e746c0ede92d89d00b3e04b297a39bd50c6bde68bd721d487cb99dcbe2af5"} Oct 30 13:07:14.865325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.367636 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45" Oct 30 13:07:14.865325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.367671 5515 scope.go:122] "RemoveContainer" containerID="84944be9206419cf73c6ad2d2b0bbe7093e02cf953f58a14fc73599e458073c0" Oct 30 13:07:14.865325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.424161 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45"] Oct 30 13:07:14.865325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.436585 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-56845b68bd-cgg45"] Oct 30 13:07:14.865325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.787445 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:07:14.865325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.787521 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:07:14.865325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.810224 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": read tcp 10.134.0.2:50078->10.134.0.129:15021: read: connection reset by peer" start-of-body= Oct 30 13:07:14.865325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.810300 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": read tcp 10.134.0.2:50078->10.134.0.129:15021: read: connection reset by peer" Oct 30 13:07:14.979391 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:14.976130 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:15.376145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.376115 5515 generic.go:356] "Generic (PLEG): container finished" podID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerID="dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62" exitCode=0 Oct 30 13:07:15.376347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.376183 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" event={"ID":"90f4bbae-0258-42ed-9d5f-95706d37e103","Type":"ContainerDied","Data":"dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62"} Oct 30 13:07:15.376662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.376644 5515 scope.go:122] "RemoveContainer" containerID="dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62" Oct 30 13:07:15.378045 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.378025 5515 generic.go:356] "Generic (PLEG): container finished" podID="117c8a18-3210-4f68-9918-0e92612dadaa" containerID="402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c" exitCode=0 Oct 30 13:07:15.378143 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.378094 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" event={"ID":"117c8a18-3210-4f68-9918-0e92612dadaa","Type":"ContainerDied","Data":"402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c"} Oct 30 13:07:15.378256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.378225 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:15.378360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.378329 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:15.378414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.378380 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:15.378476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.378459 5515 scope.go:122] "RemoveContainer" containerID="402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c" Oct 30 13:07:15.379822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.379789 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:15.379914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.379852 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:15.379914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.379886 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:15.381508 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.381487 5515 generic.go:356] "Generic (PLEG): container finished" podID="5b15247e-2417-4458-aada-9598ad342b4e" containerID="2c64d78b327475d74da78e143a2d6078b03e5a1ecfce682b528e6c4c413bd4e4" exitCode=0 Oct 30 13:07:15.381603 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.381559 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" event={"ID":"5b15247e-2417-4458-aada-9598ad342b4e","Type":"ContainerDied","Data":"2c64d78b327475d74da78e143a2d6078b03e5a1ecfce682b528e6c4c413bd4e4"} Oct 30 13:07:15.381888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.381869 5515 scope.go:122] "RemoveContainer" containerID="2c64d78b327475d74da78e143a2d6078b03e5a1ecfce682b528e6c4c413bd4e4" Oct 30 13:07:15.383935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.383905 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:15.384007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.383989 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:15.384068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:15.384040 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:16.022296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.019075 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9909d73-5c03-435c-8700-f709e8186f7a" path="/var/lib/kubelet/pods/e9909d73-5c03-435c-8700-f709e8186f7a/volumes" Oct 30 13:07:16.391537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.391501 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" event={"ID":"90f4bbae-0258-42ed-9d5f-95706d37e103","Type":"ContainerStarted","Data":"56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae"} Oct 30 13:07:16.413293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.397714 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" event={"ID":"117c8a18-3210-4f68-9918-0e92612dadaa","Type":"ContainerStarted","Data":"ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0"} Oct 30 13:07:16.431294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.427339 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" event={"ID":"5b15247e-2417-4458-aada-9598ad342b4e","Type":"ContainerStarted","Data":"5fab7c1e6fc1d129d85475fcc61658021a8dc419487052f467ab8d55df4c2e18"} Oct 30 13:07:16.550310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.544909 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:07:16.550310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.545074 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:07:16.550310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.545142 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:07:16.573307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.568501 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:07:16.573307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.569469 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:07:16.573307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.569519 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 13:07:16.574938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.574829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:16.758354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.751626 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:07:16.758354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.751762 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:07:16.758354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:16.751846 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:07:17.146309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:17.146229 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:17.546353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:17.545717 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:07:17.546353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:17.545798 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:07:17.566294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:17.559680 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:07:17.566294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:17.559754 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 13:07:17.720295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:17.719733 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:07:17.720295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:17.719799 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:07:18.071412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:18.071374 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:18.372971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:18.372883 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:18.545817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:18.545775 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:07:18.546074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:18.546043 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:07:18.565655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:18.565538 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:07:18.565655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:18.565598 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 13:07:18.719890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:18.719849 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:07:18.720254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:18.720216 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:07:19.100343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:19.086032 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:19.296300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:19.295191 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:19.548140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:19.547679 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:07:19.548140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:19.547744 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:07:19.559471 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:19.559372 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:07:19.559788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:19.559738 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 13:07:19.622333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:19.619490 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:19.722675 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:19.722638 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:07:19.722954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:19.722926 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:07:20.571332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:20.568409 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:07:20.571332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:20.568469 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:07:20.577748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:20.575116 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:07:20.577748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:20.575175 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 13:07:20.729792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:20.729754 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:07:20.730057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:20.730029 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:07:21.546792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:21.546743 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:07:21.547106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:21.547071 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:07:21.558725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:21.558686 5515 patch_prober.go:29] interesting pod/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 13:07:21.558999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:21.558969 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 13:07:21.565812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:21.565778 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:21.720305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:21.719805 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 13:07:21.720305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:21.719913 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 13:07:22.465873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.465840 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:22.572144 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.572104 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:07:22.573056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.573020 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:07:22.574204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.574180 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:07:22.589652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.589610 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:07:22.590037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.590019 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:07:22.590995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.590973 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:07:22.724298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.724176 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:07:22.724779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.724569 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:07:22.725245 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.725226 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:07:22.783428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:22.783389 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:23.005359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:23.005259 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:23.917058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:23.914657 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:24.155593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.155563 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:24.445073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.445039 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:24.609149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.608960 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc"] Oct 30 13:07:24.614946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.614138 5515 cpu_manager.go:414] "containerMap: removing container" podUID="e9909d73-5c03-435c-8700-f709e8186f7a" containerName="payload-processing" Oct 30 13:07:24.614946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.614164 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9909d73-5c03-435c-8700-f709e8186f7a" containerName="payload-processing" Oct 30 13:07:24.614946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.614377 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9909d73-5c03-435c-8700-f709e8186f7a" containerName="payload-processing" Oct 30 13:07:24.621518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.621166 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.638864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.638832 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-credential-socket\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.639046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.638990 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zjf\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-kube-api-access-76zjf\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.639133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.639093 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-data\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.639335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.639254 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-podinfo\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.639465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.639358 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-token\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.639465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.639404 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3476fa7-d001-46e8-a744-15b8c65b785a-istiod-ca-cert\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.639465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.639428 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-envoy\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.639647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.639490 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-certs\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.639647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.639543 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-socket\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.657070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.657038 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-aa466d-openshift-default-dockercfg-2zmdh\"" Oct 30 13:07:24.662090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.662049 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc"] Oct 30 13:07:24.761101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.744263 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-data\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.761483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.761447 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-podinfo\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.761635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.761617 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-token\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.761797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.761780 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3476fa7-d001-46e8-a744-15b8c65b785a-istiod-ca-cert\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.765392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.765358 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-envoy\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.765594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.765579 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-certs\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.772300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.770390 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-socket\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.772300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.770547 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-credential-socket\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.772300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.770678 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zjf\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-kube-api-access-76zjf\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.772300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.771569 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-data\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.772300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.772194 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3476fa7-d001-46e8-a744-15b8c65b785a-istiod-ca-cert\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.772903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.772477 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-socket\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.772903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.772729 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-certs\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.773019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.772980 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-credential-socket\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.801292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.796412 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-envoy\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.801292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.798102 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-podinfo\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.825304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.823187 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zjf\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-kube-api-access-76zjf\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.825304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.825131 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-token\") pod \"e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:24.842292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.840949 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:24.941492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:24.941448 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:25.191402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.191368 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:25.263447 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:07:25.263417 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3476fa7_d001_46e8_a744_15b8c65b785a.slice/crio-074d609519b1008a458d6fbb488fcb98425259770588382994baac6df8df4ca1 WatchSource:0}: Error finding container 074d609519b1008a458d6fbb488fcb98425259770588382994baac6df8df4ca1: Status 404 returned error can't find the container with id 074d609519b1008a458d6fbb488fcb98425259770588382994baac6df8df4ca1 Oct 30 13:07:25.263620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.263597 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc"] Oct 30 13:07:25.265955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.265922 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:25.266035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.265987 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:25.266035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.266019 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:25.679699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.677680 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:07:25.697442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.697404 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" event={"ID":"f3476fa7-d001-46e8-a744-15b8c65b785a","Type":"ContainerStarted","Data":"12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9"} Oct 30 13:07:25.697748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.697710 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" event={"ID":"f3476fa7-d001-46e8-a744-15b8c65b785a","Type":"ContainerStarted","Data":"074d609519b1008a458d6fbb488fcb98425259770588382994baac6df8df4ca1"} Oct 30 13:07:25.752643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.748919 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" podStartSLOduration=1.7489008350000002 podStartE2EDuration="1.748900835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:25.7467925 +0000 UTC m=+2063.873593396" watchObservedRunningTime="2026-10-30 13:07:25.748900835 +0000 UTC m=+2063.875701733" Oct 30 13:07:25.942058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:25.941958 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:26.002580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:26.002541 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:26.948550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:26.948510 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:27.227962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:27.227879 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:27.714162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:27.714127 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:27.715062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:27.715043 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:07:27.781124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:27.781075 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:28.077876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.077792 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg"] Oct 30 13:07:28.078344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.078006 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" podUID="b994b01e-7756-40c1-87df-3e026a6043f6" containerName="maas-api" containerID="cri-o://5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1" gracePeriod=30 Oct 30 13:07:28.262504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.261125 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:28.456219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.456180 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:07:28.549913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.549861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:28.597915 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.597803 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b994b01e-7756-40c1-87df-3e026a6043f6-kube-api-access-h7z7h\" (UniqueName: \"kubernetes.io/projected/b994b01e-7756-40c1-87df-3e026a6043f6-kube-api-access-h7z7h\") pod \"b994b01e-7756-40c1-87df-3e026a6043f6\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " Oct 30 13:07:28.598117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.597931 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b994b01e-7756-40c1-87df-3e026a6043f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994b01e-7756-40c1-87df-3e026a6043f6-openshift-service-ca\") pod \"b994b01e-7756-40c1-87df-3e026a6043f6\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " Oct 30 13:07:28.598117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.597998 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls\") pod \"b994b01e-7756-40c1-87df-3e026a6043f6\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " Oct 30 13:07:28.598243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.598158 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-metrics-certs\") pod \"b994b01e-7756-40c1-87df-3e026a6043f6\" (UID: \"b994b01e-7756-40c1-87df-3e026a6043f6\") " Oct 30 13:07:28.598416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.598385 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b994b01e-7756-40c1-87df-3e026a6043f6-openshift-service-ca" pod "b994b01e-7756-40c1-87df-3e026a6043f6" (UID: "b994b01e-7756-40c1-87df-3e026a6043f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:28.598760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.598743 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b994b01e-7756-40c1-87df-3e026a6043f6-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:07:28.600146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.600122 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls" pod "b994b01e-7756-40c1-87df-3e026a6043f6" (UID: "b994b01e-7756-40c1-87df-3e026a6043f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:28.600244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.600165 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b994b01e-7756-40c1-87df-3e026a6043f6-kube-api-access-h7z7h" pod "b994b01e-7756-40c1-87df-3e026a6043f6" (UID: "b994b01e-7756-40c1-87df-3e026a6043f6"). InnerVolumeSpecName "kube-api-access-h7z7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:28.600401 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.600247 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-metrics-certs" pod "b994b01e-7756-40c1-87df-3e026a6043f6" (UID: "b994b01e-7756-40c1-87df-3e026a6043f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:28.704365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.701392 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:07:28.704365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.701469 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7z7h\" (UniqueName: \"kubernetes.io/projected/b994b01e-7756-40c1-87df-3e026a6043f6-kube-api-access-h7z7h\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:07:28.704365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.701488 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b994b01e-7756-40c1-87df-3e026a6043f6-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:07:28.772640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.772604 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:28.774394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.774344 5515 generic.go:356] "Generic (PLEG): container finished" podID="b994b01e-7756-40c1-87df-3e026a6043f6" containerID="5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1" exitCode=0 Oct 30 13:07:28.774677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.774651 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" event={"ID":"b994b01e-7756-40c1-87df-3e026a6043f6","Type":"ContainerDied","Data":"5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1"} Oct 30 13:07:28.774869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.774856 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" event={"ID":"b994b01e-7756-40c1-87df-3e026a6043f6","Type":"ContainerDied","Data":"b6dbffc01e75a463727e80f49494ea15df5df739fe2113771c5be103ab4761bb"} Oct 30 13:07:28.775015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.774957 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg" Oct 30 13:07:28.775418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.774973 5515 scope.go:122] "RemoveContainer" containerID="5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1" Oct 30 13:07:28.818501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.817986 5515 scope.go:122] "RemoveContainer" containerID="5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1" Oct 30 13:07:28.822292 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:07:28.818670 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1\": container with ID starting with 5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1 not found: ID does not exist" containerID="5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1" Oct 30 13:07:28.822292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.818713 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1"} err="failed to get container status \"5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1\": rpc error: code = NotFound desc = could not find container \"5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1\": container with ID starting with 5ef10df5edf5d251a2c0950c411ddd7801e60a9ae6e0d11519280d1924e04bc1 not found: ID does not exist" Oct 30 13:07:28.840836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.840801 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg"] Oct 30 13:07:28.843554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.843522 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a121dd-778cfb458d-vd6pg"] Oct 30 13:07:28.989813 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:28.989772 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:28.990449 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:07:28.990422 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:07:29.196879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:29.196848 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:29.620498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:29.620461 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:30.001921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.001890 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b994b01e-7756-40c1-87df-3e026a6043f6" path="/var/lib/kubelet/pods/b994b01e-7756-40c1-87df-3e026a6043f6/volumes" Oct 30 13:07:30.227098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.227059 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:30.296704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.296622 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg"] Oct 30 13:07:30.297353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.297336 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b994b01e-7756-40c1-87df-3e026a6043f6" containerName="maas-api" Oct 30 13:07:30.297419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.297353 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b994b01e-7756-40c1-87df-3e026a6043f6" containerName="maas-api" Oct 30 13:07:30.297485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.297476 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b994b01e-7756-40c1-87df-3e026a6043f6" containerName="maas-api" Oct 30 13:07:30.300806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.300787 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:30.322737 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.322706 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg"] Oct 30 13:07:30.331101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.331069 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-aa466d-dockercfg-rkhrn\"" Oct 30 13:07:30.339049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.339010 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-aa466d\"" Oct 30 13:07:30.434706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.434661 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmr5v\" (UniqueName: \"kubernetes.io/projected/4375a23a-1761-49a7-bd45-8e7fe1448993-kube-api-access-pmr5v\") pod \"payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg\" (UID: \"4375a23a-1761-49a7-bd45-8e7fe1448993\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:30.434984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.434960 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4375a23a-1761-49a7-bd45-8e7fe1448993-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg\" (UID: \"4375a23a-1761-49a7-bd45-8e7fe1448993\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:30.469763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.453490 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:30.472277 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.472224 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc"] Oct 30 13:07:30.480477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.476413 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:30.483515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.483461 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc"] Oct 30 13:07:30.563034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.562255 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr5v\" (UniqueName: \"kubernetes.io/projected/4375a23a-1761-49a7-bd45-8e7fe1448993-kube-api-access-pmr5v\") pod \"payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg\" (UID: \"4375a23a-1761-49a7-bd45-8e7fe1448993\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:30.563034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.562366 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4375a23a-1761-49a7-bd45-8e7fe1448993-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg\" (UID: \"4375a23a-1761-49a7-bd45-8e7fe1448993\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:30.582500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.579639 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4375a23a-1761-49a7-bd45-8e7fe1448993-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg\" (UID: \"4375a23a-1761-49a7-bd45-8e7fe1448993\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:30.597318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.595505 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmr5v\" (UniqueName: \"kubernetes.io/projected/4375a23a-1761-49a7-bd45-8e7fe1448993-kube-api-access-pmr5v\") pod \"payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg\" (UID: \"4375a23a-1761-49a7-bd45-8e7fe1448993\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:30.615586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.615553 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:30.663670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.663633 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrf5j\" (UniqueName: \"kubernetes.io/projected/8a7d9824-153f-4aef-8faa-98223262159b-kube-api-access-vrf5j\") pod \"payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc\" (UID: \"8a7d9824-153f-4aef-8faa-98223262159b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:30.663886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.663711 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d9824-153f-4aef-8faa-98223262159b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc\" (UID: \"8a7d9824-153f-4aef-8faa-98223262159b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:30.765649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.765608 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrf5j\" (UniqueName: \"kubernetes.io/projected/8a7d9824-153f-4aef-8faa-98223262159b-kube-api-access-vrf5j\") pod \"payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc\" (UID: \"8a7d9824-153f-4aef-8faa-98223262159b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:30.765875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.765716 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d9824-153f-4aef-8faa-98223262159b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc\" (UID: \"8a7d9824-153f-4aef-8faa-98223262159b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:30.766663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.766633 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d9824-153f-4aef-8faa-98223262159b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc\" (UID: \"8a7d9824-153f-4aef-8faa-98223262159b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:30.789819 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.789782 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrf5j\" (UniqueName: \"kubernetes.io/projected/8a7d9824-153f-4aef-8faa-98223262159b-kube-api-access-vrf5j\") pod \"payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc\" (UID: \"8a7d9824-153f-4aef-8faa-98223262159b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:30.889489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.889453 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:30.925412 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:07:30.925380 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4375a23a_1761_49a7_bd45_8e7fe1448993.slice/crio-b7d42cdc125d74b877598c8a15cc9dba9bb86bb932dddcd4c2cf194806bf7126 WatchSource:0}: Error finding container b7d42cdc125d74b877598c8a15cc9dba9bb86bb932dddcd4c2cf194806bf7126: Status 404 returned error can't find the container with id b7d42cdc125d74b877598c8a15cc9dba9bb86bb932dddcd4c2cf194806bf7126 Oct 30 13:07:30.926334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:30.926307 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg"] Oct 30 13:07:31.196462 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:07:31.196432 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7d9824_153f_4aef_8faa_98223262159b.slice/crio-23775949eeddbafbc5cdf649d4191fea7b15ee548bb656e05b4dea81ff475b05 WatchSource:0}: Error finding container 23775949eeddbafbc5cdf649d4191fea7b15ee548bb656e05b4dea81ff475b05: Status 404 returned error can't find the container with id 23775949eeddbafbc5cdf649d4191fea7b15ee548bb656e05b4dea81ff475b05 Oct 30 13:07:31.196601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.196539 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc"] Oct 30 13:07:31.559509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.555617 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:31.795745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.795701 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" event={"ID":"4375a23a-1761-49a7-bd45-8e7fe1448993","Type":"ContainerStarted","Data":"b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6"} Oct 30 13:07:31.795930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.795791 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" event={"ID":"4375a23a-1761-49a7-bd45-8e7fe1448993","Type":"ContainerStarted","Data":"b7d42cdc125d74b877598c8a15cc9dba9bb86bb932dddcd4c2cf194806bf7126"} Oct 30 13:07:31.795930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.795907 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:31.797432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.797407 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" event={"ID":"8a7d9824-153f-4aef-8faa-98223262159b","Type":"ContainerStarted","Data":"dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3"} Oct 30 13:07:31.797561 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.797454 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" event={"ID":"8a7d9824-153f-4aef-8faa-98223262159b","Type":"ContainerStarted","Data":"23775949eeddbafbc5cdf649d4191fea7b15ee548bb656e05b4dea81ff475b05"} Oct 30 13:07:31.797561 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.797481 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:31.838603 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.838488 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" podStartSLOduration=1.838473782 podStartE2EDuration="1.838473782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:31.836862302 +0000 UTC m=+2069.963663202" watchObservedRunningTime="2026-10-30 13:07:31.838473782 +0000 UTC m=+2069.965274680" Oct 30 13:07:31.881295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:31.881224 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" podStartSLOduration=1.881209633 podStartE2EDuration="1.881209633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:31.877961132 +0000 UTC m=+2070.004762030" watchObservedRunningTime="2026-10-30 13:07:31.881209633 +0000 UTC m=+2070.008010531" Oct 30 13:07:32.918958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:32.918904 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:33.272375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.261108 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:33.272375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.267297 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf"] Oct 30 13:07:33.296288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.289216 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:33.350289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.347796 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf"] Oct 30 13:07:33.371322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.365665 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-76c107\"" Oct 30 13:07:33.371322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.366347 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-76c107-dockercfg-d54pb\"" Oct 30 13:07:33.371322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.367993 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923d62a-9587-4c06-a628-2a7f012e0540-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf\" (UID: \"3923d62a-9587-4c06-a628-2a7f012e0540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:33.371322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.368077 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7ll\" (UniqueName: \"kubernetes.io/projected/3923d62a-9587-4c06-a628-2a7f012e0540-kube-api-access-8h7ll\") pod \"payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf\" (UID: \"3923d62a-9587-4c06-a628-2a7f012e0540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:33.469007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.468974 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923d62a-9587-4c06-a628-2a7f012e0540-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf\" (UID: \"3923d62a-9587-4c06-a628-2a7f012e0540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:33.469207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.469033 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7ll\" (UniqueName: \"kubernetes.io/projected/3923d62a-9587-4c06-a628-2a7f012e0540-kube-api-access-8h7ll\") pod \"payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf\" (UID: \"3923d62a-9587-4c06-a628-2a7f012e0540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:33.469707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.469686 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923d62a-9587-4c06-a628-2a7f012e0540-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf\" (UID: \"3923d62a-9587-4c06-a628-2a7f012e0540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:33.493136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.493102 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7ll\" (UniqueName: \"kubernetes.io/projected/3923d62a-9587-4c06-a628-2a7f012e0540-kube-api-access-8h7ll\") pod \"payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf\" (UID: \"3923d62a-9587-4c06-a628-2a7f012e0540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:33.678726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:33.678689 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:34.019117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:34.019037 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:34.044482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:34.044452 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf"] Oct 30 13:07:34.045217 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:07:34.045194 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3923d62a_9587_4c06_a628_2a7f012e0540.slice/crio-45b7fd71499cfb4722b7c1a066f1db2017e238098e364057b9c91664550c0849 WatchSource:0}: Error finding container 45b7fd71499cfb4722b7c1a066f1db2017e238098e364057b9c91664550c0849: Status 404 returned error can't find the container with id 45b7fd71499cfb4722b7c1a066f1db2017e238098e364057b9c91664550c0849 Oct 30 13:07:34.918842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:34.918808 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" event={"ID":"3923d62a-9587-4c06-a628-2a7f012e0540","Type":"ContainerStarted","Data":"dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4"} Oct 30 13:07:34.919040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:34.918861 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" event={"ID":"3923d62a-9587-4c06-a628-2a7f012e0540","Type":"ContainerStarted","Data":"45b7fd71499cfb4722b7c1a066f1db2017e238098e364057b9c91664550c0849"} Oct 30 13:07:34.919040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:34.918910 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:34.969156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:34.969092 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" podStartSLOduration=1.969077043 podStartE2EDuration="1.969077043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:34.965490131 +0000 UTC m=+2073.092291028" watchObservedRunningTime="2026-10-30 13:07:34.969077043 +0000 UTC m=+2073.095877980" Oct 30 13:07:35.787322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:35.787256 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:37.400959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:37.400921 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:37.626503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:37.626462 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:37.857206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:37.846143 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:38.963850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:38.963034 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.667459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.667419 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.982231 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.982158 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9"] Oct 30 13:07:39.986179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.986157 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:39.992146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.992114 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-certs\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:39.992341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.992158 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-envoy\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:39.992341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.992191 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-istio-token\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:39.992432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.992356 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-socket\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:39.992432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.992407 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzx5p\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-kube-api-access-fzx5p\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:39.992574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.992464 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-credential-socket\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:39.992574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.992562 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fce36-454e-4192-92ba-04acf3135f01-istiod-ca-cert\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:39.992684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.992627 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fce36-454e-4192-92ba-04acf3135f01-istio-podinfo\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:39.992733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:39.992683 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-data\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.002530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.002506 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f9ea8f-openshift-default-dockercfg-smg9t\"" Oct 30 13:07:40.017816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.017782 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9"] Oct 30 13:07:40.093595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.093556 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-certs\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.093595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.093597 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-envoy\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.093869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.093637 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-istio-token\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.093869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.093722 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-socket\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.093869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.093764 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzx5p\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-kube-api-access-fzx5p\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.093869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.093815 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-credential-socket\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.094057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.093908 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fce36-454e-4192-92ba-04acf3135f01-istiod-ca-cert\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.094129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.094084 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-certs\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.094595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.093983 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fce36-454e-4192-92ba-04acf3135f01-istio-podinfo\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.094723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.094637 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-data\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.094793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.094744 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-socket\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.094793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.094567 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-credential-socket\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.095152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.095127 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-data\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.095323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.095302 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fce36-454e-4192-92ba-04acf3135f01-istiod-ca-cert\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.096642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.096551 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-envoy\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.105627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.097445 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fce36-454e-4192-92ba-04acf3135f01-istio-podinfo\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.117090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.117062 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-istio-token\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.117090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.117075 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzx5p\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-kube-api-access-fzx5p\") pod \"e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.141338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.141255 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:40.304977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.304872 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:40.350632 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.350598 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:40.687617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.687570 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9"] Oct 30 13:07:40.699308 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:07:40.699246 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9fce36_454e_4192_92ba_04acf3135f01.slice/crio-6337cadb9b105015789122bae0a491812ef5049ece86df263e06b2d19b8c4798 WatchSource:0}: Error finding container 6337cadb9b105015789122bae0a491812ef5049ece86df263e06b2d19b8c4798: Status 404 returned error can't find the container with id 6337cadb9b105015789122bae0a491812ef5049ece86df263e06b2d19b8c4798 Oct 30 13:07:40.703183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.702647 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:40.704710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.704669 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:40.704923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.704899 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:40.705030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.704953 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:07:40.972279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.972185 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" event={"ID":"ec9fce36-454e-4192-92ba-04acf3135f01","Type":"ContainerStarted","Data":"c966beb04a976c2a607ef367ef963cfc25b564c66ec09dd6305637d6ffff42ba"} Oct 30 13:07:40.972279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:40.972239 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" event={"ID":"ec9fce36-454e-4192-92ba-04acf3135f01","Type":"ContainerStarted","Data":"6337cadb9b105015789122bae0a491812ef5049ece86df263e06b2d19b8c4798"} Oct 30 13:07:41.026442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:41.026384 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" podStartSLOduration=2.026367224 podStartE2EDuration="2.026367224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:41.023352912 +0000 UTC m=+2079.150153810" watchObservedRunningTime="2026-10-30 13:07:41.026367224 +0000 UTC m=+2079.153168182" Oct 30 13:07:41.305930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:41.305827 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:41.312148 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:41.312113 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:41.979300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:41.979257 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:41.980389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:41.980364 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:07:42.366371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:42.366216 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:42.906057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:42.906022 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:07:42.907414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:42.907392 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:07:43.592000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:43.591958 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:43.824233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:43.824191 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:44.310125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:44.310056 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.294787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:45.294743 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.524348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:45.518089 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.927754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:45.927723 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:07:45.971314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:45.970836 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:46.183341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:46.183196 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:46.395467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:46.395433 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:47.679645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:47.679593 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:48.276165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:48.276139 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bvptt_7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca/payload-processing/0.log" Oct 30 13:07:48.508055 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:48.508019 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bvptt_7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca/payload-processing/0.log" Oct 30 13:07:49.640305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:49.640248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:50.443112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:50.443052 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:51.418418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:51.418366 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:53.281079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.281038 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt"] Oct 30 13:07:53.287235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.287210 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.318906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.318864 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt"] Oct 30 13:07:53.324817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.324783 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-aa466d-serving-cert\"" Oct 30 13:07:53.366448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.366415 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-metrics-certs\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.366657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.366474 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.366734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.366703 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda2c92-9b5b-4851-942d-a4cfe7a6499a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.366862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.366845 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tj4j\" (UniqueName: \"kubernetes.io/projected/edda2c92-9b5b-4851-942d-a4cfe7a6499a-kube-api-access-5tj4j\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.468419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.468379 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda2c92-9b5b-4851-942d-a4cfe7a6499a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.468597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.468514 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tj4j\" (UniqueName: \"kubernetes.io/projected/edda2c92-9b5b-4851-942d-a4cfe7a6499a-kube-api-access-5tj4j\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.468649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.468598 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-metrics-certs\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.468689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.468662 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.469252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.469224 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda2c92-9b5b-4851-942d-a4cfe7a6499a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.471159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.471135 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.471315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.471291 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-metrics-certs\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.484384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.484349 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:53.492732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.492702 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tj4j\" (UniqueName: \"kubernetes.io/projected/edda2c92-9b5b-4851-942d-a4cfe7a6499a-kube-api-access-5tj4j\") pod \"maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.603650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.603565 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:53.731410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.731379 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:53.937497 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:07:53.937462 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedda2c92_9b5b_4851_942d_a4cfe7a6499a.slice/crio-8dfa4907e6253bcd04a1c293af1de9ef955eb3034f8235d17e615c1d6136d2b6 WatchSource:0}: Error finding container 8dfa4907e6253bcd04a1c293af1de9ef955eb3034f8235d17e615c1d6136d2b6: Status 404 returned error can't find the container with id 8dfa4907e6253bcd04a1c293af1de9ef955eb3034f8235d17e615c1d6136d2b6 Oct 30 13:07:53.938332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:53.938306 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt"] Oct 30 13:07:54.086700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:54.086664 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" event={"ID":"edda2c92-9b5b-4851-942d-a4cfe7a6499a","Type":"ContainerStarted","Data":"8dfa4907e6253bcd04a1c293af1de9ef955eb3034f8235d17e615c1d6136d2b6"} Oct 30 13:07:55.095968 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:55.095928 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" event={"ID":"edda2c92-9b5b-4851-942d-a4cfe7a6499a","Type":"ContainerStarted","Data":"634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee"} Oct 30 13:07:55.096540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:55.096081 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:07:55.140680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:55.140629 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" podStartSLOduration=1.726948191 podStartE2EDuration="2.140614597s" totalImagesPullingTime="413.666406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:55.138840062 +0000 UTC m=+2093.265640958" watchObservedRunningTime="2026-10-30 13:07:55.140614597 +0000 UTC m=+2093.267415512" Oct 30 13:07:55.401018 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:55.400963 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:57.465152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:57.465090 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:58.989872 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:07:58.989833 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:07:59.459783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:07:59.459748 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:01.145292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:01.143298 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:01.148622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:01.148597 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:08:02.811089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:02.811048 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:04.548299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:04.543244 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:06.148091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:06.148055 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:08:06.303242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:06.303207 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:08.086199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:08.086164 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:08.386076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:08.386045 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:10.475227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:10.473754 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:10.719295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:10.719159 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:12.316930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:12.316891 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:13.932291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:13.931381 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:15.133716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.133674 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt"] Oct 30 13:08:15.134403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.133981 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" podUID="c367c5bd-d5bf-47be-881f-21427af0aad2" containerName="maas-api" containerID="cri-o://cd002dfa489a89385e47361b4d66a088c02615bde363956d7c16ee64538c6576" gracePeriod=30 Oct 30 13:08:15.281108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.281073 5515 generic.go:356] "Generic (PLEG): container finished" podID="c367c5bd-d5bf-47be-881f-21427af0aad2" containerID="cd002dfa489a89385e47361b4d66a088c02615bde363956d7c16ee64538c6576" exitCode=0 Oct 30 13:08:15.281336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.281153 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" event={"ID":"c367c5bd-d5bf-47be-881f-21427af0aad2","Type":"ContainerDied","Data":"cd002dfa489a89385e47361b4d66a088c02615bde363956d7c16ee64538c6576"} Oct 30 13:08:15.306109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.299605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:15.334641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.334259 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9"] Oct 30 13:08:15.334641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.334564 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" podUID="19dd2400-473f-4788-942c-f0ed36def00c" containerName="payload-processing" containerID="cri-o://871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9" gracePeriod=30 Oct 30 13:08:15.391336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.380500 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs"] Oct 30 13:08:15.391336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.380837 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" podUID="0263e3e3-ec51-4454-84bd-5634d45ba500" containerName="payload-pre-processing" containerID="cri-o://8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384" gracePeriod=30 Oct 30 13:08:15.514404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.514370 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:15.537332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.534421 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:08:15.593786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.593754 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-metrics-certs\") pod \"c367c5bd-d5bf-47be-881f-21427af0aad2\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " Oct 30 13:08:15.594513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.594017 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c367c5bd-d5bf-47be-881f-21427af0aad2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367c5bd-d5bf-47be-881f-21427af0aad2-openshift-service-ca\") pod \"c367c5bd-d5bf-47be-881f-21427af0aad2\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " Oct 30 13:08:15.594513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.594085 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c367c5bd-d5bf-47be-881f-21427af0aad2-kube-api-access-xbg6x\" (UniqueName: \"kubernetes.io/projected/c367c5bd-d5bf-47be-881f-21427af0aad2-kube-api-access-xbg6x\") pod \"c367c5bd-d5bf-47be-881f-21427af0aad2\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " Oct 30 13:08:15.594513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.594173 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-maas-api-tls\") pod \"c367c5bd-d5bf-47be-881f-21427af0aad2\" (UID: \"c367c5bd-d5bf-47be-881f-21427af0aad2\") " Oct 30 13:08:15.594740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.594550 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c367c5bd-d5bf-47be-881f-21427af0aad2-openshift-service-ca" pod "c367c5bd-d5bf-47be-881f-21427af0aad2" (UID: "c367c5bd-d5bf-47be-881f-21427af0aad2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:15.594874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.594852 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367c5bd-d5bf-47be-881f-21427af0aad2-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:15.607229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.603602 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c367c5bd-d5bf-47be-881f-21427af0aad2-kube-api-access-xbg6x" pod "c367c5bd-d5bf-47be-881f-21427af0aad2" (UID: "c367c5bd-d5bf-47be-881f-21427af0aad2"). InnerVolumeSpecName "kube-api-access-xbg6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:15.607229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.603601 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-metrics-certs" pod "c367c5bd-d5bf-47be-881f-21427af0aad2" (UID: "c367c5bd-d5bf-47be-881f-21427af0aad2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:15.607229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.605544 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-maas-api-tls" pod "c367c5bd-d5bf-47be-881f-21427af0aad2" (UID: "c367c5bd-d5bf-47be-881f-21427af0aad2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:15.696963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.696716 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbg6x\" (UniqueName: \"kubernetes.io/projected/c367c5bd-d5bf-47be-881f-21427af0aad2-kube-api-access-xbg6x\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:15.696963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.696750 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:15.697212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.697048 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367c5bd-d5bf-47be-881f-21427af0aad2-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:15.795291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.788195 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:15.877291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.876782 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:08:15.882292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.879311 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:08:15.917189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.917094 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0263e3e3-ec51-4454-84bd-5634d45ba500-kube-api-access-4d2km\" (UniqueName: \"kubernetes.io/projected/0263e3e3-ec51-4454-84bd-5634d45ba500-kube-api-access-4d2km\") pod \"0263e3e3-ec51-4454-84bd-5634d45ba500\" (UID: \"0263e3e3-ec51-4454-84bd-5634d45ba500\") " Oct 30 13:08:15.917569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.917543 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19dd2400-473f-4788-942c-f0ed36def00c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dd2400-473f-4788-942c-f0ed36def00c-plugins-config-volume\") pod \"19dd2400-473f-4788-942c-f0ed36def00c\" (UID: \"19dd2400-473f-4788-942c-f0ed36def00c\") " Oct 30 13:08:15.917781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.917757 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0263e3e3-ec51-4454-84bd-5634d45ba500-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0263e3e3-ec51-4454-84bd-5634d45ba500-plugins-config-volume\") pod \"0263e3e3-ec51-4454-84bd-5634d45ba500\" (UID: \"0263e3e3-ec51-4454-84bd-5634d45ba500\") " Oct 30 13:08:15.917936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.917918 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dd2400-473f-4788-942c-f0ed36def00c-kube-api-access-sg9l7\" (UniqueName: \"kubernetes.io/projected/19dd2400-473f-4788-942c-f0ed36def00c-kube-api-access-sg9l7\") pod \"19dd2400-473f-4788-942c-f0ed36def00c\" (UID: \"19dd2400-473f-4788-942c-f0ed36def00c\") " Oct 30 13:08:15.921047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.921019 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0263e3e3-ec51-4454-84bd-5634d45ba500-plugins-config-volume" pod "0263e3e3-ec51-4454-84bd-5634d45ba500" (UID: "0263e3e3-ec51-4454-84bd-5634d45ba500"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:15.922173 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.922149 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dd2400-473f-4788-942c-f0ed36def00c-plugins-config-volume" pod "19dd2400-473f-4788-942c-f0ed36def00c" (UID: "19dd2400-473f-4788-942c-f0ed36def00c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:15.926114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.925814 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dd2400-473f-4788-942c-f0ed36def00c-kube-api-access-sg9l7" pod "19dd2400-473f-4788-942c-f0ed36def00c" (UID: "19dd2400-473f-4788-942c-f0ed36def00c"). InnerVolumeSpecName "kube-api-access-sg9l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:15.928237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:15.928196 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0263e3e3-ec51-4454-84bd-5634d45ba500-kube-api-access-4d2km" pod "0263e3e3-ec51-4454-84bd-5634d45ba500" (UID: "0263e3e3-ec51-4454-84bd-5634d45ba500"). InnerVolumeSpecName "kube-api-access-4d2km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:16.020216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.020184 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d2km\" (UniqueName: \"kubernetes.io/projected/0263e3e3-ec51-4454-84bd-5634d45ba500-kube-api-access-4d2km\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:16.020396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.020219 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dd2400-473f-4788-942c-f0ed36def00c-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:16.020396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.020233 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0263e3e3-ec51-4454-84bd-5634d45ba500-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:16.020396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.020248 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg9l7\" (UniqueName: \"kubernetes.io/projected/19dd2400-473f-4788-942c-f0ed36def00c-kube-api-access-sg9l7\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:16.092071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.092028 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:16.290139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.290062 5515 generic.go:356] "Generic (PLEG): container finished" podID="19dd2400-473f-4788-942c-f0ed36def00c" containerID="871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9" exitCode=0 Oct 30 13:08:16.290139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.290127 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" Oct 30 13:08:16.290676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.290145 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" event={"ID":"19dd2400-473f-4788-942c-f0ed36def00c","Type":"ContainerDied","Data":"871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9"} Oct 30 13:08:16.290676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.290205 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9" event={"ID":"19dd2400-473f-4788-942c-f0ed36def00c","Type":"ContainerDied","Data":"51b788d0067ef530bc0fd6950dcc2d6dca77f111ec15fa36c077b540f5f40c8b"} Oct 30 13:08:16.290676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.290229 5515 scope.go:122] "RemoveContainer" containerID="871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9" Oct 30 13:08:16.291881 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.291861 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" Oct 30 13:08:16.292003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.291885 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt" event={"ID":"c367c5bd-d5bf-47be-881f-21427af0aad2","Type":"ContainerDied","Data":"f3c8500d23019f16cfda9aa2f4644594b5ce815f4bff1c562695a04c0f14ba6b"} Oct 30 13:08:16.293810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.293782 5515 generic.go:356] "Generic (PLEG): container finished" podID="0263e3e3-ec51-4454-84bd-5634d45ba500" containerID="8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384" exitCode=0 Oct 30 13:08:16.293889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.293814 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" Oct 30 13:08:16.293889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.293845 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" event={"ID":"0263e3e3-ec51-4454-84bd-5634d45ba500","Type":"ContainerDied","Data":"8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384"} Oct 30 13:08:16.293979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.293889 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs" event={"ID":"0263e3e3-ec51-4454-84bd-5634d45ba500","Type":"ContainerDied","Data":"f619109b0643aaf5c593cf3ac725e07a7ce96ef7f8eb0d087003c120dacd9da1"} Oct 30 13:08:16.304409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.304386 5515 scope.go:122] "RemoveContainer" containerID="871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9" Oct 30 13:08:16.304824 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:08:16.304769 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9\": container with ID starting with 871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9 not found: ID does not exist" containerID="871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9" Oct 30 13:08:16.304893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.304834 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9"} err="failed to get container status \"871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9\": rpc error: code = NotFound desc = could not find container \"871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9\": container with ID starting with 871d183f4a563865c565493d2b9555ab8a739b4efadf06864ea942096960ecc9 not found: ID does not exist" Oct 30 13:08:16.304893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.304854 5515 scope.go:122] "RemoveContainer" containerID="cd002dfa489a89385e47361b4d66a088c02615bde363956d7c16ee64538c6576" Oct 30 13:08:16.319102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.319069 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:16.321006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.320989 5515 scope.go:122] "RemoveContainer" containerID="8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384" Oct 30 13:08:16.333222 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.333197 5515 scope.go:122] "RemoveContainer" containerID="8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384" Oct 30 13:08:16.333618 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:08:16.333601 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384\": container with ID starting with 8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384 not found: ID does not exist" containerID="8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384" Oct 30 13:08:16.333687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.333630 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384"} err="failed to get container status \"8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384\": rpc error: code = NotFound desc = could not find container \"8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384\": container with ID starting with 8b415a9f4d22aa98881e2384b2c64a59f7df6acfc1cc816a41fd149559ada384 not found: ID does not exist" Oct 30 13:08:16.369655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.369623 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9"] Oct 30 13:08:16.382668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.382633 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa855c-694549449d-psnv9"] Oct 30 13:08:16.422358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.422324 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs"] Oct 30 13:08:16.438210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.438165 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa855c-86757bcbb7-4tnbs"] Oct 30 13:08:16.497917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.497885 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt"] Oct 30 13:08:16.514717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.514678 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa855c-57bcd9f695-mr9wt"] Oct 30 13:08:16.778955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:16.778923 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:17.305250 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:17.305217 5515 generic.go:356] "Generic (PLEG): container finished" podID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerID="12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9" exitCode=0 Oct 30 13:08:17.305871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:17.305294 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" event={"ID":"f3476fa7-d001-46e8-a744-15b8c65b785a","Type":"ContainerDied","Data":"12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9"} Oct 30 13:08:17.306010 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:17.305868 5515 scope.go:122] "RemoveContainer" containerID="12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9" Oct 30 13:08:17.307554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:17.307525 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:17.307631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:17.307588 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:17.307672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:17.307642 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:18.025322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:18.025263 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0263e3e3-ec51-4454-84bd-5634d45ba500" path="/var/lib/kubelet/pods/0263e3e3-ec51-4454-84bd-5634d45ba500/volumes" Oct 30 13:08:18.025867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:18.025844 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dd2400-473f-4788-942c-f0ed36def00c" path="/var/lib/kubelet/pods/19dd2400-473f-4788-942c-f0ed36def00c/volumes" Oct 30 13:08:18.026430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:18.026411 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c367c5bd-d5bf-47be-881f-21427af0aad2" path="/var/lib/kubelet/pods/c367c5bd-d5bf-47be-881f-21427af0aad2/volumes" Oct 30 13:08:18.325853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:18.325769 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" event={"ID":"f3476fa7-d001-46e8-a744-15b8c65b785a","Type":"ContainerStarted","Data":"76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789"} Oct 30 13:08:18.517292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:18.516835 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:18.730785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:18.730734 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:18.942432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:18.942396 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:08:18.942713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:18.942683 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 13:08:18.942760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:18.942743 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 13:08:19.942313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:19.942260 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 13:08:19.942822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:19.942338 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 13:08:20.894314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:20.890955 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:20.943876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:20.943837 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 13:08:20.944546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:20.944511 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 13:08:21.202372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:21.200653 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:21.419880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:21.419837 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:21.941900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:21.941864 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 13:08:21.942083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:21.941910 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 13:08:22.814963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:22.814899 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:22.941894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:22.941855 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 13:08:22.942083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:22.941913 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 13:08:23.905177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:23.905098 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:23.951315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:23.951248 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:08:23.952192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:23.952162 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:08:23.953143 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:23.953083 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:08:24.153856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:24.153820 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:24.392232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:24.392193 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:24.704745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:24.704642 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:25.428764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:25.428687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:27.708110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:27.704919 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:28.989832 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:08:28.989787 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:08:29.614551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:29.614507 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:31.217653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:31.217600 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:32.420841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:32.420796 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:08:32.928765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:32.928736 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:33.926205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:33.926164 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c"] Oct 30 13:08:33.978397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:33.978362 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx"] Oct 30 13:08:34.189811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:34.189727 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:34.422870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:34.422828 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:34.989680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:34.989635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:36.584860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:36.584815 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:37.770114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:37.770072 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj"] Oct 30 13:08:37.771041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:37.770988 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" podUID="77400f5c-e08f-4bb2-adaa-831c954153b7" containerName="maas-api" containerID="cri-o://9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57" gracePeriod=30 Oct 30 13:08:37.953167 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:37.953084 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:38.190190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.189655 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:38.237114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.236677 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:08:38.338874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.338728 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-maas-api-tls\") pod \"77400f5c-e08f-4bb2-adaa-831c954153b7\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " Oct 30 13:08:38.338874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.338791 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77400f5c-e08f-4bb2-adaa-831c954153b7-kube-api-access-k8mpv\" (UniqueName: \"kubernetes.io/projected/77400f5c-e08f-4bb2-adaa-831c954153b7-kube-api-access-k8mpv\") pod \"77400f5c-e08f-4bb2-adaa-831c954153b7\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " Oct 30 13:08:38.339141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.338912 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-metrics-certs\") pod \"77400f5c-e08f-4bb2-adaa-831c954153b7\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " Oct 30 13:08:38.339141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.339092 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77400f5c-e08f-4bb2-adaa-831c954153b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77400f5c-e08f-4bb2-adaa-831c954153b7-openshift-service-ca\") pod \"77400f5c-e08f-4bb2-adaa-831c954153b7\" (UID: \"77400f5c-e08f-4bb2-adaa-831c954153b7\") " Oct 30 13:08:38.339707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.339679 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77400f5c-e08f-4bb2-adaa-831c954153b7-openshift-service-ca" pod "77400f5c-e08f-4bb2-adaa-831c954153b7" (UID: "77400f5c-e08f-4bb2-adaa-831c954153b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:38.342360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.342321 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-metrics-certs" pod "77400f5c-e08f-4bb2-adaa-831c954153b7" (UID: "77400f5c-e08f-4bb2-adaa-831c954153b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:38.344763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.344740 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-maas-api-tls" pod "77400f5c-e08f-4bb2-adaa-831c954153b7" (UID: "77400f5c-e08f-4bb2-adaa-831c954153b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:38.347542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.347518 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77400f5c-e08f-4bb2-adaa-831c954153b7-kube-api-access-k8mpv" pod "77400f5c-e08f-4bb2-adaa-831c954153b7" (UID: "77400f5c-e08f-4bb2-adaa-831c954153b7"). InnerVolumeSpecName "kube-api-access-k8mpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:38.417707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.417639 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:38.446967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.446923 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:38.447386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.447363 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8mpv\" (UniqueName: \"kubernetes.io/projected/77400f5c-e08f-4bb2-adaa-831c954153b7-kube-api-access-k8mpv\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:38.447518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.447504 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77400f5c-e08f-4bb2-adaa-831c954153b7-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:38.447611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.447600 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77400f5c-e08f-4bb2-adaa-831c954153b7-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:38.627788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.627698 5515 generic.go:356] "Generic (PLEG): container finished" podID="77400f5c-e08f-4bb2-adaa-831c954153b7" containerID="9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57" exitCode=0 Oct 30 13:08:38.628191 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.628162 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" event={"ID":"77400f5c-e08f-4bb2-adaa-831c954153b7","Type":"ContainerDied","Data":"9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57"} Oct 30 13:08:38.628409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.628388 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" event={"ID":"77400f5c-e08f-4bb2-adaa-831c954153b7","Type":"ContainerDied","Data":"bf9a3a0b0d3877541db0d13462f22ba9a9c84b4fd89f66a531516e60382353b7"} Oct 30 13:08:38.628557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.628542 5515 scope.go:122] "RemoveContainer" containerID="9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57" Oct 30 13:08:38.628833 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.628815 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj" Oct 30 13:08:38.657674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.657653 5515 scope.go:122] "RemoveContainer" containerID="9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57" Oct 30 13:08:38.660214 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:08:38.659874 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57\": container with ID starting with 9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57 not found: ID does not exist" containerID="9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57" Oct 30 13:08:38.660214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.659916 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57"} err="failed to get container status \"9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57\": rpc error: code = NotFound desc = could not find container \"9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57\": container with ID starting with 9eaa8ab594a500244297f2f4c248e268807ed53e76a06b0e18701c1006e82b57 not found: ID does not exist" Oct 30 13:08:38.724297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.723325 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj"] Oct 30 13:08:38.730091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.729581 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8c35df-578f9fcfd6-r2qkj"] Oct 30 13:08:38.934314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.932340 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" podUID="a60c9615-42b8-47bd-9547-3abb6e39171d" containerName="payload-processing" containerID="cri-o://d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3" gracePeriod=25 Oct 30 13:08:38.981861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.981797 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" podUID="9d34394d-ac4a-4bc1-9cbb-5b2c6d215587" containerName="payload-pre-processing" containerID="cri-o://249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2" gracePeriod=25 Oct 30 13:08:38.982600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:38.982572 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:39.339755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.339724 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:08:39.445049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.445017 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:08:39.473696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.473660 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60c9615-42b8-47bd-9547-3abb6e39171d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60c9615-42b8-47bd-9547-3abb6e39171d-plugins-config-volume\") pod \"a60c9615-42b8-47bd-9547-3abb6e39171d\" (UID: \"a60c9615-42b8-47bd-9547-3abb6e39171d\") " Oct 30 13:08:39.473890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.473860 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60c9615-42b8-47bd-9547-3abb6e39171d-kube-api-access-zdxrf\" (UniqueName: \"kubernetes.io/projected/a60c9615-42b8-47bd-9547-3abb6e39171d-kube-api-access-zdxrf\") pod \"a60c9615-42b8-47bd-9547-3abb6e39171d\" (UID: \"a60c9615-42b8-47bd-9547-3abb6e39171d\") " Oct 30 13:08:39.474204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.474171 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60c9615-42b8-47bd-9547-3abb6e39171d-plugins-config-volume" pod "a60c9615-42b8-47bd-9547-3abb6e39171d" (UID: "a60c9615-42b8-47bd-9547-3abb6e39171d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:39.475658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.475057 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60c9615-42b8-47bd-9547-3abb6e39171d-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:39.480901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.480866 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60c9615-42b8-47bd-9547-3abb6e39171d-kube-api-access-zdxrf" pod "a60c9615-42b8-47bd-9547-3abb6e39171d" (UID: "a60c9615-42b8-47bd-9547-3abb6e39171d"). InnerVolumeSpecName "kube-api-access-zdxrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:39.576152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.576078 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-plugins-config-volume\") pod \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\" (UID: \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\") " Oct 30 13:08:39.576333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.576279 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-kube-api-access-rgswx\" (UniqueName: \"kubernetes.io/projected/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-kube-api-access-rgswx\") pod \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\" (UID: \"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587\") " Oct 30 13:08:39.576537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.576512 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-plugins-config-volume" pod "9d34394d-ac4a-4bc1-9cbb-5b2c6d215587" (UID: "9d34394d-ac4a-4bc1-9cbb-5b2c6d215587"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:39.576970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.576955 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxrf\" (UniqueName: \"kubernetes.io/projected/a60c9615-42b8-47bd-9547-3abb6e39171d-kube-api-access-zdxrf\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:39.577007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.576974 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:39.578377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.578323 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-kube-api-access-rgswx" pod "9d34394d-ac4a-4bc1-9cbb-5b2c6d215587" (UID: "9d34394d-ac4a-4bc1-9cbb-5b2c6d215587"). InnerVolumeSpecName "kube-api-access-rgswx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:39.636660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.636620 5515 generic.go:356] "Generic (PLEG): container finished" podID="9d34394d-ac4a-4bc1-9cbb-5b2c6d215587" containerID="249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2" exitCode=0 Oct 30 13:08:39.636867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.636688 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" Oct 30 13:08:39.636867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.636711 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" event={"ID":"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587","Type":"ContainerDied","Data":"249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2"} Oct 30 13:08:39.636867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.636772 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx" event={"ID":"9d34394d-ac4a-4bc1-9cbb-5b2c6d215587","Type":"ContainerDied","Data":"c70062608e29ac0598226ce9ea2210262b8f992e13919ae8b683c1c31fecbdb3"} Oct 30 13:08:39.636867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.636802 5515 scope.go:122] "RemoveContainer" containerID="249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2" Oct 30 13:08:39.638468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.638438 5515 generic.go:356] "Generic (PLEG): container finished" podID="a60c9615-42b8-47bd-9547-3abb6e39171d" containerID="d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3" exitCode=0 Oct 30 13:08:39.638592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.638485 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" event={"ID":"a60c9615-42b8-47bd-9547-3abb6e39171d","Type":"ContainerDied","Data":"d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3"} Oct 30 13:08:39.638592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.638507 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" Oct 30 13:08:39.638592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.638551 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c" event={"ID":"a60c9615-42b8-47bd-9547-3abb6e39171d","Type":"ContainerDied","Data":"917d9117efbbde3fb9d3e17a56287584e8ca8b58e63b3f93b4092ba3ed63854e"} Oct 30 13:08:39.684476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.678245 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgswx\" (UniqueName: \"kubernetes.io/projected/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587-kube-api-access-rgswx\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:39.708253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.708218 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:39.709462 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.709427 5515 scope.go:122] "RemoveContainer" containerID="249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2" Oct 30 13:08:39.709951 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:08:39.709922 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2\": container with ID starting with 249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2 not found: ID does not exist" containerID="249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2" Oct 30 13:08:39.710097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.709960 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2"} err="failed to get container status \"249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2\": rpc error: code = NotFound desc = could not find container \"249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2\": container with ID starting with 249ed50116086311c356ff336767f41729d1466acf11af41618c573f50b039a2 not found: ID does not exist" Oct 30 13:08:39.710097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.709987 5515 scope.go:122] "RemoveContainer" containerID="d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3" Oct 30 13:08:39.722333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.722305 5515 scope.go:122] "RemoveContainer" containerID="d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3" Oct 30 13:08:39.722681 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:08:39.722661 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3\": container with ID starting with d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3 not found: ID does not exist" containerID="d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3" Oct 30 13:08:39.722815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.722692 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3"} err="failed to get container status \"d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3\": rpc error: code = NotFound desc = could not find container \"d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3\": container with ID starting with d4914142086dc656755a2e0da70817ef6ee3a4ac8929bf44610cb1f120f71ad3 not found: ID does not exist" Oct 30 13:08:39.736164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.736133 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c"] Oct 30 13:08:39.750736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.750705 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4a97e9-677745666b-rgc9c"] Oct 30 13:08:39.787571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.787526 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx"] Oct 30 13:08:39.800714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:39.800681 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4a97e9-7b4d9bd8c9-8wgdx"] Oct 30 13:08:40.002214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:40.002178 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77400f5c-e08f-4bb2-adaa-831c954153b7" path="/var/lib/kubelet/pods/77400f5c-e08f-4bb2-adaa-831c954153b7/volumes" Oct 30 13:08:40.002663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:40.002648 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d34394d-ac4a-4bc1-9cbb-5b2c6d215587" path="/var/lib/kubelet/pods/9d34394d-ac4a-4bc1-9cbb-5b2c6d215587/volumes" Oct 30 13:08:40.003040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:40.003027 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60c9615-42b8-47bd-9547-3abb6e39171d" path="/var/lib/kubelet/pods/a60c9615-42b8-47bd-9547-3abb6e39171d/volumes" Oct 30 13:08:40.051852 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:40.051809 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:40.255935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:40.255847 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:40.781560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:40.779861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.012526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.012490 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.248515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.248479 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.457060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.457025 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk"] Oct 30 13:08:41.458078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458055 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c367c5bd-d5bf-47be-881f-21427af0aad2" containerName="maas-api" Oct 30 13:08:41.458134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458084 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c367c5bd-d5bf-47be-881f-21427af0aad2" containerName="maas-api" Oct 30 13:08:41.458134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458116 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0263e3e3-ec51-4454-84bd-5634d45ba500" containerName="payload-pre-processing" Oct 30 13:08:41.458134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458124 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0263e3e3-ec51-4454-84bd-5634d45ba500" containerName="payload-pre-processing" Oct 30 13:08:41.458239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458144 5515 cpu_manager.go:414] "containerMap: removing container" podUID="a60c9615-42b8-47bd-9547-3abb6e39171d" containerName="payload-processing" Oct 30 13:08:41.458239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458152 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60c9615-42b8-47bd-9547-3abb6e39171d" containerName="payload-processing" Oct 30 13:08:41.458239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458168 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9d34394d-ac4a-4bc1-9cbb-5b2c6d215587" containerName="payload-pre-processing" Oct 30 13:08:41.458239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458175 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34394d-ac4a-4bc1-9cbb-5b2c6d215587" containerName="payload-pre-processing" Oct 30 13:08:41.458239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458190 5515 cpu_manager.go:414] "containerMap: removing container" podUID="19dd2400-473f-4788-942c-f0ed36def00c" containerName="payload-processing" Oct 30 13:08:41.458239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458197 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dd2400-473f-4788-942c-f0ed36def00c" containerName="payload-processing" Oct 30 13:08:41.458239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458233 5515 cpu_manager.go:414] "containerMap: removing container" podUID="77400f5c-e08f-4bb2-adaa-831c954153b7" containerName="maas-api" Oct 30 13:08:41.458239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458240 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77400f5c-e08f-4bb2-adaa-831c954153b7" containerName="maas-api" Oct 30 13:08:41.458495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458445 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="77400f5c-e08f-4bb2-adaa-831c954153b7" containerName="maas-api" Oct 30 13:08:41.458495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458468 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c367c5bd-d5bf-47be-881f-21427af0aad2" containerName="maas-api" Oct 30 13:08:41.458495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458481 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0263e3e3-ec51-4454-84bd-5634d45ba500" containerName="payload-pre-processing" Oct 30 13:08:41.458590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458496 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60c9615-42b8-47bd-9547-3abb6e39171d" containerName="payload-processing" Oct 30 13:08:41.458590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458508 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34394d-ac4a-4bc1-9cbb-5b2c6d215587" containerName="payload-pre-processing" Oct 30 13:08:41.458590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.458518 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dd2400-473f-4788-942c-f0ed36def00c" containerName="payload-processing" Oct 30 13:08:41.462732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.462705 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.504410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.501401 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d84f47-openshift-default-dockercfg-rq52w\"" Oct 30 13:08:41.507182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.507150 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk"] Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.596500 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.611356 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fddc62-e822-4b6f-be6f-870ac7691dac-istiod-ca-cert\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.611490 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-token\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.611529 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-socket\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.611588 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-credential-socket\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.611618 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-envoy\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.611644 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-data\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.611672 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-certs\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.611713 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-podinfo\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.614973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.611859 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dkb\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-kube-api-access-n5dkb\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.720963 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-token\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.721026 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-socket\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.721078 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-credential-socket\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.721107 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-envoy\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.721131 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-data\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.721159 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-certs\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.721199 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-podinfo\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.721360 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dkb\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-kube-api-access-n5dkb\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.721445 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fddc62-e822-4b6f-be6f-870ac7691dac-istiod-ca-cert\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.722298 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fddc62-e822-4b6f-be6f-870ac7691dac-istiod-ca-cert\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.722885 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-socket\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.723121 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-credential-socket\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.723692 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-certs\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.730292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.723937 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-data\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.760256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.756801 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-podinfo\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.777253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.777178 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dkb\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-kube-api-access-n5dkb\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.792043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.788113 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-token\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.792043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.791932 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-envoy\") pod \"e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:41.825288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:41.825213 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:42.079100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:42.079007 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:42.474979 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:08:42.474947 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fddc62_e822_4b6f_be6f_870ac7691dac.slice/crio-427a6d97fc0211d00e9e317555abf5aca758598c6d869de0bc5419e6375896c1 WatchSource:0}: Error finding container 427a6d97fc0211d00e9e317555abf5aca758598c6d869de0bc5419e6375896c1: Status 404 returned error can't find the container with id 427a6d97fc0211d00e9e317555abf5aca758598c6d869de0bc5419e6375896c1 Oct 30 13:08:42.475575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:42.475529 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk"] Oct 30 13:08:42.478152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:42.477594 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:42.478152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:42.477682 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:42.478152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:42.477720 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:42.721346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:42.717101 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:42.776834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:42.776799 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" event={"ID":"37fddc62-e822-4b6f-be6f-870ac7691dac","Type":"ContainerStarted","Data":"d249bd733c6ff50d0d1efb871a47e5016a5190ad126e5bf77522d751711b5505"} Oct 30 13:08:42.777029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:42.776852 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" event={"ID":"37fddc62-e822-4b6f-be6f-870ac7691dac","Type":"ContainerStarted","Data":"427a6d97fc0211d00e9e317555abf5aca758598c6d869de0bc5419e6375896c1"} Oct 30 13:08:42.828243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:42.828194 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" podStartSLOduration=1.828179682 podStartE2EDuration="1.828179682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:42.824877328 +0000 UTC m=+2140.951678226" watchObservedRunningTime="2026-10-30 13:08:42.828179682 +0000 UTC m=+2140.954980580" Oct 30 13:08:43.050501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:43.050407 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:08:43.050501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:43.050468 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.079421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:43.079384 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:43.087609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:43.087578 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:43.783692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:43.783647 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:43.784682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:43.784657 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:08:44.283070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:44.283028 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:44.504092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:44.504042 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:44.775294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:44.773015 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:44.867311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:44.867259 5515 generic.go:356] "Generic (PLEG): container finished" podID="ec9fce36-454e-4192-92ba-04acf3135f01" containerID="c966beb04a976c2a607ef367ef963cfc25b564c66ec09dd6305637d6ffff42ba" exitCode=0 Oct 30 13:08:44.867482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:44.867390 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" event={"ID":"ec9fce36-454e-4192-92ba-04acf3135f01","Type":"ContainerDied","Data":"c966beb04a976c2a607ef367ef963cfc25b564c66ec09dd6305637d6ffff42ba"} Oct 30 13:08:44.868239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:44.868218 5515 scope.go:122] "RemoveContainer" containerID="c966beb04a976c2a607ef367ef963cfc25b564c66ec09dd6305637d6ffff42ba" Oct 30 13:08:44.869964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:44.869930 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:44.870062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:44.870008 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:44.870062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:44.870039 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:45.003736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:45.003674 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:45.430456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:45.430424 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:45.715306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:45.715195 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:45.875447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:45.875411 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" event={"ID":"ec9fce36-454e-4192-92ba-04acf3135f01","Type":"ContainerStarted","Data":"642a7cb45a8c2527191554489c677c5871a4d78f7e109e69427dc252f5ec3f25"} Oct 30 13:08:45.973305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:45.973200 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:46.307797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:46.307712 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:08:46.308048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:46.308009 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:08:46.308299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:46.308249 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:08:47.319532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.319484 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:08:47.320014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.319560 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:08:47.560887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.560849 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:47.668842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.668806 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l"] Oct 30 13:08:47.672129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.672108 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:08:47.731362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.731321 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l"] Oct 30 13:08:47.744700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.744658 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd6c2b6-791c-4133-9033-2ced2194d2a2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l\" (UID: \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:08:47.745201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.745171 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqdrn\" (UniqueName: \"kubernetes.io/projected/0bd6c2b6-791c-4133-9033-2ced2194d2a2-kube-api-access-vqdrn\") pod \"payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l\" (UID: \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:08:47.811587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.811550 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d84f47\"" Oct 30 13:08:47.811859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.811841 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d84f47-dockercfg-jnnl7\"" Oct 30 13:08:47.903289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.877549 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:48.048171 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:47.904766 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqdrn\" (UniqueName: \"kubernetes.io/projected/0bd6c2b6-791c-4133-9033-2ced2194d2a2-kube-api-access-vqdrn\") pod \"payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l\" (UID: \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:08:48.048687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:48.048655 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd6c2b6-791c-4133-9033-2ced2194d2a2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l\" (UID: \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:08:48.050825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:48.050805 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd6c2b6-791c-4133-9033-2ced2194d2a2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l\" (UID: \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:08:48.056323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:48.056294 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqdrn\" (UniqueName: \"kubernetes.io/projected/0bd6c2b6-791c-4133-9033-2ced2194d2a2-kube-api-access-vqdrn\") pod \"payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l\" (UID: \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:08:48.258389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:48.258344 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:48.286132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:48.286105 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:08:48.313749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:48.313666 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:08:48.313998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:48.313964 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:08:48.631501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:48.631464 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l"] Oct 30 13:08:48.631826 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:08:48.631778 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd6c2b6_791c_4133_9033_2ced2194d2a2.slice/crio-bc3d15f72e1c3aee198d061f5b72c4f4127893413fa2ff0f52b36603303ba1c9 WatchSource:0}: Error finding container bc3d15f72e1c3aee198d061f5b72c4f4127893413fa2ff0f52b36603303ba1c9: Status 404 returned error can't find the container with id bc3d15f72e1c3aee198d061f5b72c4f4127893413fa2ff0f52b36603303ba1c9 Oct 30 13:08:49.098294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:49.097738 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" event={"ID":"0bd6c2b6-791c-4133-9033-2ced2194d2a2","Type":"ContainerStarted","Data":"1183d6d09f16ec9aec006a7baa348d2309f1fff98b4bb9c6c1a611e143d798b0"} Oct 30 13:08:49.098294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:49.097810 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" event={"ID":"0bd6c2b6-791c-4133-9033-2ced2194d2a2","Type":"ContainerStarted","Data":"bc3d15f72e1c3aee198d061f5b72c4f4127893413fa2ff0f52b36603303ba1c9"} Oct 30 13:08:49.104601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:49.104519 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:08:49.153924 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:49.153871 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" podStartSLOduration=2.153856541 podStartE2EDuration="2.153856541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:49.151453071 +0000 UTC m=+2147.278253970" watchObservedRunningTime="2026-10-30 13:08:49.153856541 +0000 UTC m=+2147.280657458" Oct 30 13:08:49.309203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:49.308679 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:08:49.309203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:49.308760 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:08:49.622336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:49.617858 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:49.916293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:49.909391 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:50.160025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:50.151703 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:50.315209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:50.312794 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:08:50.315209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:50.312855 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:08:50.389842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:50.389803 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:51.379661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:51.379615 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:08:51.380352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:51.380320 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:08:51.381910 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:51.381883 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:08:52.479416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:52.479378 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:52.718206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:52.718141 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:52.874329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:52.874230 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn"] Oct 30 13:08:52.874558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:52.874520 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" containerID="cri-o://56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae" gracePeriod=30 Oct 30 13:08:53.105077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:53.105041 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:53.841720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:53.841681 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs"] Oct 30 13:08:53.842556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:53.842508 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" podUID="ce79343e-e824-44f4-b5ea-acb39dcdab89" containerName="maas-api" containerID="cri-o://53cf18f865c499d68a6ce5abd452202d44560f4ca4add20c5a73fa77b3fb1c5d" gracePeriod=30 Oct 30 13:08:53.998940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:53.998901 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:54.334146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.334106 5515 generic.go:356] "Generic (PLEG): container finished" podID="ce79343e-e824-44f4-b5ea-acb39dcdab89" containerID="53cf18f865c499d68a6ce5abd452202d44560f4ca4add20c5a73fa77b3fb1c5d" exitCode=0 Oct 30 13:08:54.334677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.334409 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" event={"ID":"ce79343e-e824-44f4-b5ea-acb39dcdab89","Type":"ContainerDied","Data":"53cf18f865c499d68a6ce5abd452202d44560f4ca4add20c5a73fa77b3fb1c5d"} Oct 30 13:08:54.334892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.334874 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" event={"ID":"ce79343e-e824-44f4-b5ea-acb39dcdab89","Type":"ContainerDied","Data":"25a5577d896a16bad4f3f7d37161eec239fd99313f9caf88a42570088732f60a"} Oct 30 13:08:54.335013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.334997 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a5577d896a16bad4f3f7d37161eec239fd99313f9caf88a42570088732f60a" Oct 30 13:08:54.339478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.339451 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:08:54.355893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.355853 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-metrics-certs\") pod \"ce79343e-e824-44f4-b5ea-acb39dcdab89\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " Oct 30 13:08:54.356197 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.356174 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce79343e-e824-44f4-b5ea-acb39dcdab89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce79343e-e824-44f4-b5ea-acb39dcdab89-openshift-service-ca\") pod \"ce79343e-e824-44f4-b5ea-acb39dcdab89\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " Oct 30 13:08:54.356550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.356526 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce79343e-e824-44f4-b5ea-acb39dcdab89-kube-api-access-rnqkl\" (UniqueName: \"kubernetes.io/projected/ce79343e-e824-44f4-b5ea-acb39dcdab89-kube-api-access-rnqkl\") pod \"ce79343e-e824-44f4-b5ea-acb39dcdab89\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " Oct 30 13:08:54.356783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.356763 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-maas-api-tls\") pod \"ce79343e-e824-44f4-b5ea-acb39dcdab89\" (UID: \"ce79343e-e824-44f4-b5ea-acb39dcdab89\") " Oct 30 13:08:54.411296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.409160 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce79343e-e824-44f4-b5ea-acb39dcdab89-openshift-service-ca" pod "ce79343e-e824-44f4-b5ea-acb39dcdab89" (UID: "ce79343e-e824-44f4-b5ea-acb39dcdab89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:54.428125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.428084 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-metrics-certs" pod "ce79343e-e824-44f4-b5ea-acb39dcdab89" (UID: "ce79343e-e824-44f4-b5ea-acb39dcdab89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:54.428449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.428406 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-maas-api-tls" pod "ce79343e-e824-44f4-b5ea-acb39dcdab89" (UID: "ce79343e-e824-44f4-b5ea-acb39dcdab89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:54.428951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.428901 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce79343e-e824-44f4-b5ea-acb39dcdab89-kube-api-access-rnqkl" pod "ce79343e-e824-44f4-b5ea-acb39dcdab89" (UID: "ce79343e-e824-44f4-b5ea-acb39dcdab89"). InnerVolumeSpecName "kube-api-access-rnqkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:54.458693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.458586 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.458693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.458629 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce79343e-e824-44f4-b5ea-acb39dcdab89-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.458693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.458645 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnqkl\" (UniqueName: \"kubernetes.io/projected/ce79343e-e824-44f4-b5ea-acb39dcdab89-kube-api-access-rnqkl\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.458693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.458658 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce79343e-e824-44f4-b5ea-acb39dcdab89-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:54.581375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.581337 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:54.900307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:54.900259 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:55.132095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.132061 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c"] Oct 30 13:08:55.133166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.133144 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ce79343e-e824-44f4-b5ea-acb39dcdab89" containerName="maas-api" Oct 30 13:08:55.133297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.133169 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce79343e-e824-44f4-b5ea-acb39dcdab89" containerName="maas-api" Oct 30 13:08:55.133543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.133519 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce79343e-e824-44f4-b5ea-acb39dcdab89" containerName="maas-api" Oct 30 13:08:55.139768 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.139748 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.162079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.162016 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c"] Oct 30 13:08:55.169702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.169661 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5ba2ef-openshift-default-dockercfg-9wsmv\"" Oct 30 13:08:55.181701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.181668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:55.184253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.184220 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-podinfo\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.184409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.184297 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-token\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.184482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.184426 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-socket\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.184543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.184519 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-envoy\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.184666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.184640 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08af005-2812-4d6c-97d0-c7aea99250bd-istiod-ca-cert\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.184762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.184741 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lrk\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-kube-api-access-n5lrk\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.184868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.184851 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-credential-socket\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.184925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.184910 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-data\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.185360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.185021 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-certs\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.285868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.285834 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-socket\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.285900 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-envoy\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.285965 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08af005-2812-4d6c-97d0-c7aea99250bd-istiod-ca-cert\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286015 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lrk\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-kube-api-access-n5lrk\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286197 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286094 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-credential-socket\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286197 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286126 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-data\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286216 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-certs\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286312 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-podinfo\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286472 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286353 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-token\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286489 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-credential-socket\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286580 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-data\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286635 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-socket\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286748 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-certs\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.286906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.286788 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08af005-2812-4d6c-97d0-c7aea99250bd-istiod-ca-cert\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.288794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.288767 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-envoy\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.289331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.289311 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-podinfo\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.310548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.310515 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lrk\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-kube-api-access-n5lrk\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.310970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.310947 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-token\") pod \"e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.435349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.434754 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:55.435349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.435218 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" Oct 30 13:08:55.460291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.459592 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:55.539324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.538842 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs"] Oct 30 13:08:55.543649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.543617 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs"] Oct 30 13:08:55.649722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.649686 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:55.840349 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:08:55.840316 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08af005_2812_4d6c_97d0_c7aea99250bd.slice/crio-727020f48713a81223606ae906bc72d1719e2682e55a3517bd7e46f9dcbb82bc WatchSource:0}: Error finding container 727020f48713a81223606ae906bc72d1719e2682e55a3517bd7e46f9dcbb82bc: Status 404 returned error can't find the container with id 727020f48713a81223606ae906bc72d1719e2682e55a3517bd7e46f9dcbb82bc Oct 30 13:08:55.840595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.840571 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c"] Oct 30 13:08:55.843057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.843021 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:55.843146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.843089 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:55.843146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:55.843117 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:08:56.006491 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:56.006460 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce79343e-e824-44f4-b5ea-acb39dcdab89" path="/var/lib/kubelet/pods/ce79343e-e824-44f4-b5ea-acb39dcdab89/volumes" Oct 30 13:08:56.443328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:56.443279 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" event={"ID":"c08af005-2812-4d6c-97d0-c7aea99250bd","Type":"ContainerStarted","Data":"0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500"} Oct 30 13:08:56.443556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:56.443346 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" event={"ID":"c08af005-2812-4d6c-97d0-c7aea99250bd","Type":"ContainerStarted","Data":"727020f48713a81223606ae906bc72d1719e2682e55a3517bd7e46f9dcbb82bc"} Oct 30 13:08:56.460485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:56.460451 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:56.466000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:56.465977 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:56.497907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:56.497840 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" podStartSLOduration=1.497820924 podStartE2EDuration="1.497820924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:56.49341457 +0000 UTC m=+2154.620215493" watchObservedRunningTime="2026-10-30 13:08:56.497820924 +0000 UTC m=+2154.624621824" Oct 30 13:08:56.800921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:56.800829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:57.028637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:57.028596 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4a97e9-d87788dd-wd9vs" podUID="ce79343e-e824-44f4-b5ea-acb39dcdab89" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.133:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.450051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:57.450015 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:57.451103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:57.451084 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:08:58.167692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.167624 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:08:58.289093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289056 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-credential-socket\") pod \"90f4bbae-0258-42ed-9d5f-95706d37e103\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " Oct 30 13:08:58.289339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289146 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-envoy\") pod \"90f4bbae-0258-42ed-9d5f-95706d37e103\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " Oct 30 13:08:58.289339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289219 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-token\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-token\") pod \"90f4bbae-0258-42ed-9d5f-95706d37e103\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " Oct 30 13:08:58.289339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289301 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f4bbae-0258-42ed-9d5f-95706d37e103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f4bbae-0258-42ed-9d5f-95706d37e103-istiod-ca-cert\") pod \"90f4bbae-0258-42ed-9d5f-95706d37e103\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " Oct 30 13:08:58.289521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289368 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-podinfo\") pod \"90f4bbae-0258-42ed-9d5f-95706d37e103\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " Oct 30 13:08:58.289521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289450 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-kube-api-access-shj7s\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-kube-api-access-shj7s\") pod \"90f4bbae-0258-42ed-9d5f-95706d37e103\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " Oct 30 13:08:58.289521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289468 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-credential-socket" pod "90f4bbae-0258-42ed-9d5f-95706d37e103" (UID: "90f4bbae-0258-42ed-9d5f-95706d37e103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:58.289667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289521 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-data\") pod \"90f4bbae-0258-42ed-9d5f-95706d37e103\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " Oct 30 13:08:58.289667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289560 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-socket\") pod \"90f4bbae-0258-42ed-9d5f-95706d37e103\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " Oct 30 13:08:58.289667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289589 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-certs\") pod \"90f4bbae-0258-42ed-9d5f-95706d37e103\" (UID: \"90f4bbae-0258-42ed-9d5f-95706d37e103\") " Oct 30 13:08:58.289868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289841 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-socket" pod "90f4bbae-0258-42ed-9d5f-95706d37e103" (UID: "90f4bbae-0258-42ed-9d5f-95706d37e103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:58.289936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.289886 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f4bbae-0258-42ed-9d5f-95706d37e103-istiod-ca-cert" pod "90f4bbae-0258-42ed-9d5f-95706d37e103" (UID: "90f4bbae-0258-42ed-9d5f-95706d37e103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:58.290161 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.290132 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-data" pod "90f4bbae-0258-42ed-9d5f-95706d37e103" (UID: "90f4bbae-0258-42ed-9d5f-95706d37e103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:58.290161 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.290148 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-certs" pod "90f4bbae-0258-42ed-9d5f-95706d37e103" (UID: "90f4bbae-0258-42ed-9d5f-95706d37e103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:58.290400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.290379 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:58.290450 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.290408 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:58.290450 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.290425 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:58.290450 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.290438 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f4bbae-0258-42ed-9d5f-95706d37e103-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:58.290547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.290453 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:58.291751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.291719 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-podinfo" pod "90f4bbae-0258-42ed-9d5f-95706d37e103" (UID: "90f4bbae-0258-42ed-9d5f-95706d37e103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:08:58.291910 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.291808 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-kube-api-access-shj7s" pod "90f4bbae-0258-42ed-9d5f-95706d37e103" (UID: "90f4bbae-0258-42ed-9d5f-95706d37e103"). InnerVolumeSpecName "kube-api-access-shj7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:58.291978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.291912 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-envoy" pod "90f4bbae-0258-42ed-9d5f-95706d37e103" (UID: "90f4bbae-0258-42ed-9d5f-95706d37e103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:58.292016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.291993 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-token" pod "90f4bbae-0258-42ed-9d5f-95706d37e103" (UID: "90f4bbae-0258-42ed-9d5f-95706d37e103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:58.391855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.391815 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:58.391855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.391852 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:58.391855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.391860 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f4bbae-0258-42ed-9d5f-95706d37e103-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:58.392124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.391870 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shj7s\" (UniqueName: \"kubernetes.io/projected/90f4bbae-0258-42ed-9d5f-95706d37e103-kube-api-access-shj7s\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:08:58.458529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.458437 5515 generic.go:356] "Generic (PLEG): container finished" podID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerID="56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae" exitCode=0 Oct 30 13:08:58.458529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.458492 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" Oct 30 13:08:58.458772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.458522 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" event={"ID":"90f4bbae-0258-42ed-9d5f-95706d37e103","Type":"ContainerDied","Data":"56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae"} Oct 30 13:08:58.458772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.458577 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn" event={"ID":"90f4bbae-0258-42ed-9d5f-95706d37e103","Type":"ContainerDied","Data":"90289b357f4002d4ad2eb4e1d86d7a963e372d403d65fc6768e23dc3c4855d11"} Oct 30 13:08:58.458772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.458600 5515 scope.go:122] "RemoveContainer" containerID="56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae" Oct 30 13:08:58.470228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.470205 5515 scope.go:122] "RemoveContainer" containerID="dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62" Oct 30 13:08:58.480616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.480587 5515 scope.go:122] "RemoveContainer" containerID="56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae" Oct 30 13:08:58.480958 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:08:58.480938 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae\": container with ID starting with 56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae not found: ID does not exist" containerID="56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae" Oct 30 13:08:58.481028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.480969 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae"} err="failed to get container status \"56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae\": rpc error: code = NotFound desc = could not find container \"56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae\": container with ID starting with 56d3772880e9801e4511d6d3c530dbe98326dfe39e5c91a80b4bfa8e4ae251ae not found: ID does not exist" Oct 30 13:08:58.481028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.480988 5515 scope.go:122] "RemoveContainer" containerID="dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62" Oct 30 13:08:58.481298 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:08:58.481261 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62\": container with ID starting with dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62 not found: ID does not exist" containerID="dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62" Oct 30 13:08:58.481405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.481304 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62"} err="failed to get container status \"dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62\": rpc error: code = NotFound desc = could not find container \"dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62\": container with ID starting with dc83ffec8c4deb26600e51f2fdef588479b0ea8a354fc7c0bb8650d28ca6cf62 not found: ID does not exist" Oct 30 13:08:58.511794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.511755 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn"] Oct 30 13:08:58.524475 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.524430 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e0e74-openshift-default-74dbf5d47-rwqkn"] Oct 30 13:08:58.815708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:58.815627 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:58.989176 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:08:58.989136 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:08:59.084882 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:59.084792 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:59.301759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:08:59.301732 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:00.002704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:00.002671 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" path="/var/lib/kubelet/pods/90f4bbae-0258-42ed-9d5f-95706d37e103/volumes" Oct 30 13:09:00.172516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:00.172487 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:09:00.772429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:00.772395 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:00.988600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:00.988565 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:01.034889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.034805 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb"] Oct 30 13:09:01.035516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.035499 5515 cpu_manager.go:414] "containerMap: removing container" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" Oct 30 13:09:01.035574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.035517 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" Oct 30 13:09:01.035574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.035530 5515 cpu_manager.go:414] "containerMap: removing container" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" Oct 30 13:09:01.035574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.035535 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" Oct 30 13:09:01.035678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.035625 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" Oct 30 13:09:01.039390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.039365 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:01.057192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.057165 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5ba2ef\"" Oct 30 13:09:01.070252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.070222 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5ba2ef-dockercfg-ckkg2\"" Oct 30 13:09:01.070602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.070579 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb"] Oct 30 13:09:01.114605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.114563 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt"] Oct 30 13:09:01.116079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.116050 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f4bbae-0258-42ed-9d5f-95706d37e103" containerName="istio-proxy" Oct 30 13:09:01.318312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.318184 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:01.351243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.351199 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrknl\" (UniqueName: \"kubernetes.io/projected/ce00f683-b3db-4b8a-8734-61b877751893-kube-api-access-nrknl\") pod \"payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb\" (UID: \"ce00f683-b3db-4b8a-8734-61b877751893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:01.359470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.359429 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce00f683-b3db-4b8a-8734-61b877751893-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb\" (UID: \"ce00f683-b3db-4b8a-8734-61b877751893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:01.372634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.372597 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:01.372937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.372919 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt"] Oct 30 13:09:01.461085 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.461045 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce00f683-b3db-4b8a-8734-61b877751893-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb\" (UID: \"ce00f683-b3db-4b8a-8734-61b877751893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:01.461340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.461107 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t76tf\" (UniqueName: \"kubernetes.io/projected/3f7eb4d7-8164-4074-be07-de089f528ca7-kube-api-access-t76tf\") pod \"payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt\" (UID: \"3f7eb4d7-8164-4074-be07-de089f528ca7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:01.461340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.461286 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrknl\" (UniqueName: \"kubernetes.io/projected/ce00f683-b3db-4b8a-8734-61b877751893-kube-api-access-nrknl\") pod \"payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb\" (UID: \"ce00f683-b3db-4b8a-8734-61b877751893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:01.461475 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.461351 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eb4d7-8164-4074-be07-de089f528ca7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt\" (UID: \"3f7eb4d7-8164-4074-be07-de089f528ca7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:01.461896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.461873 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce00f683-b3db-4b8a-8734-61b877751893-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb\" (UID: \"ce00f683-b3db-4b8a-8734-61b877751893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:01.499494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.499462 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrknl\" (UniqueName: \"kubernetes.io/projected/ce00f683-b3db-4b8a-8734-61b877751893-kube-api-access-nrknl\") pod \"payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb\" (UID: \"ce00f683-b3db-4b8a-8734-61b877751893\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:01.563029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.562992 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eb4d7-8164-4074-be07-de089f528ca7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt\" (UID: \"3f7eb4d7-8164-4074-be07-de089f528ca7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:01.563208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.563092 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t76tf\" (UniqueName: \"kubernetes.io/projected/3f7eb4d7-8164-4074-be07-de089f528ca7-kube-api-access-t76tf\") pod \"payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt\" (UID: \"3f7eb4d7-8164-4074-be07-de089f528ca7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:01.563831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.563804 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eb4d7-8164-4074-be07-de089f528ca7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt\" (UID: \"3f7eb4d7-8164-4074-be07-de089f528ca7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:01.609591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.609517 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t76tf\" (UniqueName: \"kubernetes.io/projected/3f7eb4d7-8164-4074-be07-de089f528ca7-kube-api-access-t76tf\") pod \"payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt\" (UID: \"3f7eb4d7-8164-4074-be07-de089f528ca7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:01.653466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.653427 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:01.688190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.688150 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:01.980584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:01.980557 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb"] Oct 30 13:09:01.980934 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:09:01.980663 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce00f683_b3db_4b8a_8734_61b877751893.slice/crio-dc63e8e8b51a6c399dec3de77d019446482e6e0713b4247387be9267e431deab WatchSource:0}: Error finding container dc63e8e8b51a6c399dec3de77d019446482e6e0713b4247387be9267e431deab: Status 404 returned error can't find the container with id dc63e8e8b51a6c399dec3de77d019446482e6e0713b4247387be9267e431deab Oct 30 13:09:02.048529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.048501 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt"] Oct 30 13:09:02.048834 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:09:02.048811 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7eb4d7_8164_4074_be07_de089f528ca7.slice/crio-4b383c56175182cb11c1cdf1ce57f92e00d59f5b71646d9c0033c5a32c6256d3 WatchSource:0}: Error finding container 4b383c56175182cb11c1cdf1ce57f92e00d59f5b71646d9c0033c5a32c6256d3: Status 404 returned error can't find the container with id 4b383c56175182cb11c1cdf1ce57f92e00d59f5b71646d9c0033c5a32c6256d3 Oct 30 13:09:02.493572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.493536 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" event={"ID":"3f7eb4d7-8164-4074-be07-de089f528ca7","Type":"ContainerStarted","Data":"222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22"} Oct 30 13:09:02.493779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.493606 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:02.493779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.493630 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" event={"ID":"3f7eb4d7-8164-4074-be07-de089f528ca7","Type":"ContainerStarted","Data":"4b383c56175182cb11c1cdf1ce57f92e00d59f5b71646d9c0033c5a32c6256d3"} Oct 30 13:09:02.495148 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.495121 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" event={"ID":"ce00f683-b3db-4b8a-8734-61b877751893","Type":"ContainerStarted","Data":"a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92"} Oct 30 13:09:02.495244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.495178 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:02.495244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.495198 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" event={"ID":"ce00f683-b3db-4b8a-8734-61b877751893","Type":"ContainerStarted","Data":"dc63e8e8b51a6c399dec3de77d019446482e6e0713b4247387be9267e431deab"} Oct 30 13:09:02.551310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.551239 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" podStartSLOduration=1.551224857 podStartE2EDuration="1.551224857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:02.549046912 +0000 UTC m=+2160.675847810" watchObservedRunningTime="2026-10-30 13:09:02.551224857 +0000 UTC m=+2160.678025782" Oct 30 13:09:02.614396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.614340 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" podStartSLOduration=2.614324796 podStartE2EDuration="2.614324796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:02.611787302 +0000 UTC m=+2160.738588199" watchObservedRunningTime="2026-10-30 13:09:02.614324796 +0000 UTC m=+2160.741125694" Oct 30 13:09:02.637072 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.637027 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc"] Oct 30 13:09:02.683735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.683702 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg"] Oct 30 13:09:02.958288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:02.949963 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:03.177209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:03.177173 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:03.502818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:03.502725 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:04.738299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:04.737010 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:05.003840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:05.003749 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:05.058713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:05.058674 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb"] Oct 30 13:09:05.058915 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:05.058892 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" containerID="cri-o://ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0" gracePeriod=30 Oct 30 13:09:05.276057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:05.275952 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:05.501731 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:05.501696 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:07.394087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:07.394048 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:07.638490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:07.638390 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" podUID="8a7d9824-153f-4aef-8faa-98223262159b" containerName="payload-processing" containerID="cri-o://dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3" gracePeriod=25 Oct 30 13:09:07.684369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:07.684324 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" podUID="4375a23a-1761-49a7-bd45-8e7fe1448993" containerName="payload-pre-processing" containerID="cri-o://b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6" gracePeriod=25 Oct 30 13:09:07.726863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:07.726824 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:09:07.727018 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:07.726877 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:09:07.972002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:07.971974 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:09:08.016973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.016946 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:09:08.107084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.107047 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4375a23a-1761-49a7-bd45-8e7fe1448993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4375a23a-1761-49a7-bd45-8e7fe1448993-plugins-config-volume\") pod \"4375a23a-1761-49a7-bd45-8e7fe1448993\" (UID: \"4375a23a-1761-49a7-bd45-8e7fe1448993\") " Oct 30 13:09:08.107331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.107301 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7d9824-153f-4aef-8faa-98223262159b-kube-api-access-vrf5j\" (UniqueName: \"kubernetes.io/projected/8a7d9824-153f-4aef-8faa-98223262159b-kube-api-access-vrf5j\") pod \"8a7d9824-153f-4aef-8faa-98223262159b\" (UID: \"8a7d9824-153f-4aef-8faa-98223262159b\") " Oct 30 13:09:08.107431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.107374 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7d9824-153f-4aef-8faa-98223262159b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d9824-153f-4aef-8faa-98223262159b-plugins-config-volume\") pod \"8a7d9824-153f-4aef-8faa-98223262159b\" (UID: \"8a7d9824-153f-4aef-8faa-98223262159b\") " Oct 30 13:09:08.107431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.107420 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4375a23a-1761-49a7-bd45-8e7fe1448993-kube-api-access-pmr5v\" (UniqueName: \"kubernetes.io/projected/4375a23a-1761-49a7-bd45-8e7fe1448993-kube-api-access-pmr5v\") pod \"4375a23a-1761-49a7-bd45-8e7fe1448993\" (UID: \"4375a23a-1761-49a7-bd45-8e7fe1448993\") " Oct 30 13:09:08.107540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.107463 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4375a23a-1761-49a7-bd45-8e7fe1448993-plugins-config-volume" pod "4375a23a-1761-49a7-bd45-8e7fe1448993" (UID: "4375a23a-1761-49a7-bd45-8e7fe1448993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:08.107765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.107740 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7d9824-153f-4aef-8faa-98223262159b-plugins-config-volume" pod "8a7d9824-153f-4aef-8faa-98223262159b" (UID: "8a7d9824-153f-4aef-8faa-98223262159b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:08.108058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.108022 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d9824-153f-4aef-8faa-98223262159b-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:08.108123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.108062 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4375a23a-1761-49a7-bd45-8e7fe1448993-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:08.109696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.109672 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7d9824-153f-4aef-8faa-98223262159b-kube-api-access-vrf5j" pod "8a7d9824-153f-4aef-8faa-98223262159b" (UID: "8a7d9824-153f-4aef-8faa-98223262159b"). InnerVolumeSpecName "kube-api-access-vrf5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:08.110049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.110032 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4375a23a-1761-49a7-bd45-8e7fe1448993-kube-api-access-pmr5v" pod "4375a23a-1761-49a7-bd45-8e7fe1448993" (UID: "4375a23a-1761-49a7-bd45-8e7fe1448993"). InnerVolumeSpecName "kube-api-access-pmr5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:08.210218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.210127 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrf5j\" (UniqueName: \"kubernetes.io/projected/8a7d9824-153f-4aef-8faa-98223262159b-kube-api-access-vrf5j\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:08.210218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.210167 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmr5v\" (UniqueName: \"kubernetes.io/projected/4375a23a-1761-49a7-bd45-8e7fe1448993-kube-api-access-pmr5v\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:08.539389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.539287 5515 generic.go:356] "Generic (PLEG): container finished" podID="4375a23a-1761-49a7-bd45-8e7fe1448993" containerID="b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6" exitCode=0 Oct 30 13:09:08.539389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.539346 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" Oct 30 13:09:08.539389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.539370 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" event={"ID":"4375a23a-1761-49a7-bd45-8e7fe1448993","Type":"ContainerDied","Data":"b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6"} Oct 30 13:09:08.539983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.539425 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg" event={"ID":"4375a23a-1761-49a7-bd45-8e7fe1448993","Type":"ContainerDied","Data":"b7d42cdc125d74b877598c8a15cc9dba9bb86bb932dddcd4c2cf194806bf7126"} Oct 30 13:09:08.539983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.539451 5515 scope.go:122] "RemoveContainer" containerID="b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6" Oct 30 13:09:08.541093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.541068 5515 generic.go:356] "Generic (PLEG): container finished" podID="8a7d9824-153f-4aef-8faa-98223262159b" containerID="dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3" exitCode=0 Oct 30 13:09:08.541196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.541107 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" event={"ID":"8a7d9824-153f-4aef-8faa-98223262159b","Type":"ContainerDied","Data":"dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3"} Oct 30 13:09:08.541196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.541115 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" Oct 30 13:09:08.541196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.541152 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc" event={"ID":"8a7d9824-153f-4aef-8faa-98223262159b","Type":"ContainerDied","Data":"23775949eeddbafbc5cdf649d4191fea7b15ee548bb656e05b4dea81ff475b05"} Oct 30 13:09:08.551080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.551060 5515 scope.go:122] "RemoveContainer" containerID="b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6" Oct 30 13:09:08.551520 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:09:08.551484 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6\": container with ID starting with b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6 not found: ID does not exist" containerID="b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6" Oct 30 13:09:08.551634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.551531 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6"} err="failed to get container status \"b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6\": rpc error: code = NotFound desc = could not find container \"b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6\": container with ID starting with b53576039556b8e9b10ab8b8df8c772fc0c76b09f293a8bdf77a23231e87bee6 not found: ID does not exist" Oct 30 13:09:08.551634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.551557 5515 scope.go:122] "RemoveContainer" containerID="dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3" Oct 30 13:09:08.561550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.561530 5515 scope.go:122] "RemoveContainer" containerID="dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3" Oct 30 13:09:08.561859 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:09:08.561836 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3\": container with ID starting with dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3 not found: ID does not exist" containerID="dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3" Oct 30 13:09:08.561929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.561870 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3"} err="failed to get container status \"dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3\": rpc error: code = NotFound desc = could not find container \"dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3\": container with ID starting with dac9fc7c50235ce9f1fe5ebdeb59127531a09ecc835710197f3accf9bce4d4d3 not found: ID does not exist" Oct 30 13:09:08.603635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.600125 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg"] Oct 30 13:09:08.612418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.612383 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-aa466d-7d69848679-w2mcg"] Oct 30 13:09:08.661419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.661389 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc"] Oct 30 13:09:08.664147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.664126 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-aa466d-6d4d675cf8-6gplc"] Oct 30 13:09:08.992163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.992119 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr"] Oct 30 13:09:08.992850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.992835 5515 cpu_manager.go:414] "containerMap: removing container" podUID="8a7d9824-153f-4aef-8faa-98223262159b" containerName="payload-processing" Oct 30 13:09:08.992901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.992851 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7d9824-153f-4aef-8faa-98223262159b" containerName="payload-processing" Oct 30 13:09:08.992901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.992868 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4375a23a-1761-49a7-bd45-8e7fe1448993" containerName="payload-pre-processing" Oct 30 13:09:08.992901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.992874 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4375a23a-1761-49a7-bd45-8e7fe1448993" containerName="payload-pre-processing" Oct 30 13:09:08.992997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.992987 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7d9824-153f-4aef-8faa-98223262159b" containerName="payload-processing" Oct 30 13:09:08.993034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.993000 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4375a23a-1761-49a7-bd45-8e7fe1448993" containerName="payload-pre-processing" Oct 30 13:09:08.998064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:08.998044 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.017101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.017054 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-d84f47-serving-cert\"" Oct 30 13:09:09.020318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.020289 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr"] Oct 30 13:09:09.121797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.121761 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rdz\" (UniqueName: \"kubernetes.io/projected/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-kube-api-access-n5rdz\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.121979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.121884 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.122087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.122066 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-metrics-certs\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.122131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.122103 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.170002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.169951 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:09.223114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.223077 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rdz\" (UniqueName: \"kubernetes.io/projected/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-kube-api-access-n5rdz\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.223328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.223174 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.223390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.223355 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-metrics-certs\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.223390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.223383 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.223906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.223876 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.225821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.225797 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-metrics-certs\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.225923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.225861 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.269335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.261737 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rdz\" (UniqueName: \"kubernetes.io/projected/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-kube-api-access-n5rdz\") pod \"maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.316292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.312661 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:09.406231 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.406202 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:09.651705 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:09.651681 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr"] Oct 30 13:09:09.653852 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:09:09.653822 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd36fae0_af4b_46c6_ac38_99740e6bd1c7.slice/crio-c75ab476add17df2e5b869d77a96ec71702773e260b97bbe315a67802a302f6b WatchSource:0}: Error finding container c75ab476add17df2e5b869d77a96ec71702773e260b97bbe315a67802a302f6b: Status 404 returned error can't find the container with id c75ab476add17df2e5b869d77a96ec71702773e260b97bbe315a67802a302f6b Oct 30 13:09:10.002150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.002065 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4375a23a-1761-49a7-bd45-8e7fe1448993" path="/var/lib/kubelet/pods/4375a23a-1761-49a7-bd45-8e7fe1448993/volumes" Oct 30 13:09:10.002573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.002557 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7d9824-153f-4aef-8faa-98223262159b" path="/var/lib/kubelet/pods/8a7d9824-153f-4aef-8faa-98223262159b/volumes" Oct 30 13:09:10.393421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.393394 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:09:10.466729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.466694 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-istio-token\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-istio-token\") pod \"117c8a18-3210-4f68-9918-0e92612dadaa\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " Oct 30 13:09:10.466938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.466746 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-data\") pod \"117c8a18-3210-4f68-9918-0e92612dadaa\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " Oct 30 13:09:10.466938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.466833 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117c8a18-3210-4f68-9918-0e92612dadaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117c8a18-3210-4f68-9918-0e92612dadaa-istiod-ca-cert\") pod \"117c8a18-3210-4f68-9918-0e92612dadaa\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " Oct 30 13:09:10.466938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.466898 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/117c8a18-3210-4f68-9918-0e92612dadaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117c8a18-3210-4f68-9918-0e92612dadaa-istio-podinfo\") pod \"117c8a18-3210-4f68-9918-0e92612dadaa\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " Oct 30 13:09:10.467121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.466943 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-certs\") pod \"117c8a18-3210-4f68-9918-0e92612dadaa\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " Oct 30 13:09:10.467121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.467092 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-kube-api-access-jrch4\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-kube-api-access-jrch4\") pod \"117c8a18-3210-4f68-9918-0e92612dadaa\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " Oct 30 13:09:10.467232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.467218 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-envoy\") pod \"117c8a18-3210-4f68-9918-0e92612dadaa\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " Oct 30 13:09:10.467307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.467091 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-data" pod "117c8a18-3210-4f68-9918-0e92612dadaa" (UID: "117c8a18-3210-4f68-9918-0e92612dadaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:10.467307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.467230 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-certs" pod "117c8a18-3210-4f68-9918-0e92612dadaa" (UID: "117c8a18-3210-4f68-9918-0e92612dadaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:10.467307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.467258 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117c8a18-3210-4f68-9918-0e92612dadaa-istiod-ca-cert" pod "117c8a18-3210-4f68-9918-0e92612dadaa" (UID: "117c8a18-3210-4f68-9918-0e92612dadaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:10.467307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.467286 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-socket\") pod \"117c8a18-3210-4f68-9918-0e92612dadaa\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " Oct 30 13:09:10.467502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.467364 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-credential-socket\") pod \"117c8a18-3210-4f68-9918-0e92612dadaa\" (UID: \"117c8a18-3210-4f68-9918-0e92612dadaa\") " Oct 30 13:09:10.467557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.467509 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-socket" pod "117c8a18-3210-4f68-9918-0e92612dadaa" (UID: "117c8a18-3210-4f68-9918-0e92612dadaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:10.467637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.467611 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-credential-socket" pod "117c8a18-3210-4f68-9918-0e92612dadaa" (UID: "117c8a18-3210-4f68-9918-0e92612dadaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:10.468196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.468169 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.468406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.468201 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117c8a18-3210-4f68-9918-0e92612dadaa-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.468406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.468216 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.468406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.468230 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.468406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.468242 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.469705 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.469671 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-kube-api-access-jrch4" pod "117c8a18-3210-4f68-9918-0e92612dadaa" (UID: "117c8a18-3210-4f68-9918-0e92612dadaa"). InnerVolumeSpecName "kube-api-access-jrch4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:10.469907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.469887 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-envoy" pod "117c8a18-3210-4f68-9918-0e92612dadaa" (UID: "117c8a18-3210-4f68-9918-0e92612dadaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:10.469973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.469906 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/117c8a18-3210-4f68-9918-0e92612dadaa-istio-podinfo" pod "117c8a18-3210-4f68-9918-0e92612dadaa" (UID: "117c8a18-3210-4f68-9918-0e92612dadaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:09:10.470049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.470034 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-istio-token" pod "117c8a18-3210-4f68-9918-0e92612dadaa" (UID: "117c8a18-3210-4f68-9918-0e92612dadaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:10.552483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.552384 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt"] Oct 30 13:09:10.552666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.552591 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" podUID="edda2c92-9b5b-4851-942d-a4cfe7a6499a" containerName="maas-api" containerID="cri-o://634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee" gracePeriod=30 Oct 30 13:09:10.563092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.562772 5515 generic.go:356] "Generic (PLEG): container finished" podID="117c8a18-3210-4f68-9918-0e92612dadaa" containerID="ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0" exitCode=0 Oct 30 13:09:10.563092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.562825 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" Oct 30 13:09:10.563092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.562908 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" event={"ID":"117c8a18-3210-4f68-9918-0e92612dadaa","Type":"ContainerDied","Data":"ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0"} Oct 30 13:09:10.563092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.562950 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb" event={"ID":"117c8a18-3210-4f68-9918-0e92612dadaa","Type":"ContainerDied","Data":"8ccf2a373c3f3e33c9206e28294b4287919196c0db74be7aa61bbc32280d3841"} Oct 30 13:09:10.563092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.562977 5515 scope.go:122] "RemoveContainer" containerID="ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0" Oct 30 13:09:10.568464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.567513 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" event={"ID":"dd36fae0-af4b-46c6-ac38-99740e6bd1c7","Type":"ContainerStarted","Data":"3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76"} Oct 30 13:09:10.568464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.567580 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" event={"ID":"dd36fae0-af4b-46c6-ac38-99740e6bd1c7","Type":"ContainerStarted","Data":"c75ab476add17df2e5b869d77a96ec71702773e260b97bbe315a67802a302f6b"} Oct 30 13:09:10.568464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.568395 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:10.569446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.569237 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117c8a18-3210-4f68-9918-0e92612dadaa-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.569446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.569369 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.569446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.569390 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117c8a18-3210-4f68-9918-0e92612dadaa-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.569446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.569423 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrch4\" (UniqueName: \"kubernetes.io/projected/117c8a18-3210-4f68-9918-0e92612dadaa-kube-api-access-jrch4\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.627931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.627907 5515 scope.go:122] "RemoveContainer" containerID="402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c" Oct 30 13:09:10.670170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.670146 5515 scope.go:122] "RemoveContainer" containerID="ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0" Oct 30 13:09:10.670657 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:09:10.670631 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0\": container with ID starting with ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0 not found: ID does not exist" containerID="ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0" Oct 30 13:09:10.670742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.670666 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0"} err="failed to get container status \"ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0\": rpc error: code = NotFound desc = could not find container \"ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0\": container with ID starting with ab3306af911bfb967cebde95c60d6b71d11bbbcf0311359a4c607bffd6cedad0 not found: ID does not exist" Oct 30 13:09:10.670742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.670685 5515 scope.go:122] "RemoveContainer" containerID="402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c" Oct 30 13:09:10.671045 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:09:10.671014 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c\": container with ID starting with 402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c not found: ID does not exist" containerID="402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c" Oct 30 13:09:10.671150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.671051 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c"} err="failed to get container status \"402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c\": rpc error: code = NotFound desc = could not find container \"402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c\": container with ID starting with 402e50a0376ea00710493972b0af089ec7c6e18a5e54d9c0cd544c6bfd7e490c not found: ID does not exist" Oct 30 13:09:10.712758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.712723 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb"] Oct 30 13:09:10.713805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.713778 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:10.722336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.722300 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4a97e9-openshift-default-b9f55f6d5-86znb"] Oct 30 13:09:10.826621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.826569 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" podStartSLOduration=2.394498603 podStartE2EDuration="2.826554366s" totalImagesPullingTime="432.055763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:10.824569336 +0000 UTC m=+2168.951370235" watchObservedRunningTime="2026-10-30 13:09:10.826554366 +0000 UTC m=+2168.953355263" Oct 30 13:09:10.842735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.842711 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:09:10.935936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.935893 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:10.974971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.974934 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edda2c92-9b5b-4851-942d-a4cfe7a6499a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda2c92-9b5b-4851-942d-a4cfe7a6499a-openshift-service-ca\") pod \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " Oct 30 13:09:10.974971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.974972 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-maas-api-tls\") pod \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " Oct 30 13:09:10.975180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.975113 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-metrics-certs\") pod \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " Oct 30 13:09:10.975352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.975320 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edda2c92-9b5b-4851-942d-a4cfe7a6499a-kube-api-access-5tj4j\" (UniqueName: \"kubernetes.io/projected/edda2c92-9b5b-4851-942d-a4cfe7a6499a-kube-api-access-5tj4j\") pod \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\" (UID: \"edda2c92-9b5b-4851-942d-a4cfe7a6499a\") " Oct 30 13:09:10.975448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.975425 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edda2c92-9b5b-4851-942d-a4cfe7a6499a-openshift-service-ca" pod "edda2c92-9b5b-4851-942d-a4cfe7a6499a" (UID: "edda2c92-9b5b-4851-942d-a4cfe7a6499a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:10.975817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.975799 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda2c92-9b5b-4851-942d-a4cfe7a6499a-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:10.977345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.977315 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-maas-api-tls" pod "edda2c92-9b5b-4851-942d-a4cfe7a6499a" (UID: "edda2c92-9b5b-4851-942d-a4cfe7a6499a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:10.977468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.977347 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edda2c92-9b5b-4851-942d-a4cfe7a6499a-kube-api-access-5tj4j" pod "edda2c92-9b5b-4851-942d-a4cfe7a6499a" (UID: "edda2c92-9b5b-4851-942d-a4cfe7a6499a"). InnerVolumeSpecName "kube-api-access-5tj4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:10.977468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:10.977365 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-metrics-certs" pod "edda2c92-9b5b-4851-942d-a4cfe7a6499a" (UID: "edda2c92-9b5b-4851-942d-a4cfe7a6499a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:11.116390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.094183 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:11.127349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.127318 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda2c92-9b5b-4851-942d-a4cfe7a6499a-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:11.131542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.131507 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tj4j\" (UniqueName: \"kubernetes.io/projected/edda2c92-9b5b-4851-942d-a4cfe7a6499a-kube-api-access-5tj4j\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:11.539042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.539010 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:11.577296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.577247 5515 generic.go:356] "Generic (PLEG): container finished" podID="edda2c92-9b5b-4851-942d-a4cfe7a6499a" containerID="634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee" exitCode=0 Oct 30 13:09:11.577500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.577331 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" Oct 30 13:09:11.577500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.577343 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" event={"ID":"edda2c92-9b5b-4851-942d-a4cfe7a6499a","Type":"ContainerDied","Data":"634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee"} Oct 30 13:09:11.577500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.577410 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt" event={"ID":"edda2c92-9b5b-4851-942d-a4cfe7a6499a","Type":"ContainerDied","Data":"8dfa4907e6253bcd04a1c293af1de9ef955eb3034f8235d17e615c1d6136d2b6"} Oct 30 13:09:11.577500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.577443 5515 scope.go:122] "RemoveContainer" containerID="634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee" Oct 30 13:09:11.608477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.608456 5515 scope.go:122] "RemoveContainer" containerID="634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee" Oct 30 13:09:11.608840 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:09:11.608815 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee\": container with ID starting with 634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee not found: ID does not exist" containerID="634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee" Oct 30 13:09:11.608916 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.608849 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee"} err="failed to get container status \"634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee\": rpc error: code = NotFound desc = could not find container \"634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee\": container with ID starting with 634d821d492af2b988f8555bcf821a934eb16e5c87d2209a242f0bb7d41b8aee not found: ID does not exist" Oct 30 13:09:11.647692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.647655 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt"] Oct 30 13:09:11.659026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.658993 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-aa466d-797f5485b-pp8dt"] Oct 30 13:09:11.743428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:11.743392 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:12.002131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:12.002029 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" path="/var/lib/kubelet/pods/117c8a18-3210-4f68-9918-0e92612dadaa/volumes" Oct 30 13:09:12.002712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:12.002687 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edda2c92-9b5b-4851-942d-a4cfe7a6499a" path="/var/lib/kubelet/pods/edda2c92-9b5b-4851-942d-a4cfe7a6499a/volumes" Oct 30 13:09:12.195781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:12.195749 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:13.505424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:13.505395 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:09:13.505827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:13.505474 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:09:14.333310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:14.328145 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:16.380218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:16.380149 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:17.460411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:17.460374 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:17.592592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:17.592564 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:09:17.690101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:17.690049 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:18.009639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:18.009608 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:18.997798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:18.997764 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:19.227091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:19.227053 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:19.243139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:19.243105 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc"] Oct 30 13:09:19.243396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:19.243350 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" containerID="cri-o://76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789" gracePeriod=30 Oct 30 13:09:19.454911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:19.454876 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:19.679447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:19.679418 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:20.939543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.939506 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd"] Oct 30 13:09:20.940162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.940146 5515 cpu_manager.go:414] "containerMap: removing container" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" Oct 30 13:09:20.940215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.940163 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" Oct 30 13:09:20.940215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.940176 5515 cpu_manager.go:414] "containerMap: removing container" podUID="edda2c92-9b5b-4851-942d-a4cfe7a6499a" containerName="maas-api" Oct 30 13:09:20.940215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.940181 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edda2c92-9b5b-4851-942d-a4cfe7a6499a" containerName="maas-api" Oct 30 13:09:20.940215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.940194 5515 cpu_manager.go:414] "containerMap: removing container" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" Oct 30 13:09:20.940215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.940198 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" Oct 30 13:09:20.940453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.940339 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" Oct 30 13:09:20.940453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.940348 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="edda2c92-9b5b-4851-942d-a4cfe7a6499a" containerName="maas-api" Oct 30 13:09:20.940453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.940356 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="117c8a18-3210-4f68-9918-0e92612dadaa" containerName="istio-proxy" Oct 30 13:09:20.945247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.945226 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:20.968477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.968447 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-5ba2ef-serving-cert\"" Oct 30 13:09:20.978309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:20.972833 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd"] Oct 30 13:09:21.102104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.102066 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.102347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.102209 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-metrics-certs\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.102347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.102251 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6gw\" (UniqueName: \"kubernetes.io/projected/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-kube-api-access-ql6gw\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.102347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.102318 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.127483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.127451 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:21.239345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.229046 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.239345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.230026 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.239345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.230365 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-metrics-certs\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.239345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.230425 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6gw\" (UniqueName: \"kubernetes.io/projected/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-kube-api-access-ql6gw\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.239345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.230452 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.261375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.258135 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6gw\" (UniqueName: \"kubernetes.io/projected/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-kube-api-access-ql6gw\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.261375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.259680 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-metrics-certs\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.264506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.264460 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.329964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.329906 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:21.560399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.560316 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:21.866524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.866476 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd"] Oct 30 13:09:21.867692 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:09:21.867666 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7f38b2_319f_425e_aeb6_f465c3ec22ad.slice/crio-1e9a526cdc65065924de131b38b9ccc00c6bd460bc1ca8a0648da4a17973383a WatchSource:0}: Error finding container 1e9a526cdc65065924de131b38b9ccc00c6bd460bc1ca8a0648da4a17973383a: Status 404 returned error can't find the container with id 1e9a526cdc65065924de131b38b9ccc00c6bd460bc1ca8a0648da4a17973383a Oct 30 13:09:21.894022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:21.893992 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:22.133894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:22.130105 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:22.434345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:22.432874 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:09:22.656095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:22.656060 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" event={"ID":"2e7f38b2-319f-425e-aeb6-f465c3ec22ad","Type":"ContainerStarted","Data":"f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8"} Oct 30 13:09:22.656332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:22.656114 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" event={"ID":"2e7f38b2-319f-425e-aeb6-f465c3ec22ad","Type":"ContainerStarted","Data":"1e9a526cdc65065924de131b38b9ccc00c6bd460bc1ca8a0648da4a17973383a"} Oct 30 13:09:22.656332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:22.656260 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:22.701905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:22.701792 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" podStartSLOduration=2.348000791 podStartE2EDuration="2.701775281s" totalImagesPullingTime="353.77449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:22.701351961 +0000 UTC m=+2180.828152858" watchObservedRunningTime="2026-10-30 13:09:22.701775281 +0000 UTC m=+2180.828576191" Oct 30 13:09:22.973988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:22.973887 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:23.955064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:23.955023 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:09:23.955574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:23.955086 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:09:24.563960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.563930 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:09:24.627680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.627605 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-token\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-token\") pod \"f3476fa7-d001-46e8-a744-15b8c65b785a\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " Oct 30 13:09:24.627680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.627673 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-data\") pod \"f3476fa7-d001-46e8-a744-15b8c65b785a\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " Oct 30 13:09:24.627946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.627767 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-envoy\") pod \"f3476fa7-d001-46e8-a744-15b8c65b785a\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " Oct 30 13:09:24.627946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.627805 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-kube-api-access-76zjf\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-kube-api-access-76zjf\") pod \"f3476fa7-d001-46e8-a744-15b8c65b785a\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " Oct 30 13:09:24.627946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.627858 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-certs\") pod \"f3476fa7-d001-46e8-a744-15b8c65b785a\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " Oct 30 13:09:24.627946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.627894 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-socket\") pod \"f3476fa7-d001-46e8-a744-15b8c65b785a\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " Oct 30 13:09:24.627946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.627926 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-podinfo\") pod \"f3476fa7-d001-46e8-a744-15b8c65b785a\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " Oct 30 13:09:24.628184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.628140 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-credential-socket\") pod \"f3476fa7-d001-46e8-a744-15b8c65b785a\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " Oct 30 13:09:24.628184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.628169 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3476fa7-d001-46e8-a744-15b8c65b785a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3476fa7-d001-46e8-a744-15b8c65b785a-istiod-ca-cert\") pod \"f3476fa7-d001-46e8-a744-15b8c65b785a\" (UID: \"f3476fa7-d001-46e8-a744-15b8c65b785a\") " Oct 30 13:09:24.628841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.628795 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-data" pod "f3476fa7-d001-46e8-a744-15b8c65b785a" (UID: "f3476fa7-d001-46e8-a744-15b8c65b785a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.629093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.629057 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3476fa7-d001-46e8-a744-15b8c65b785a-istiod-ca-cert" pod "f3476fa7-d001-46e8-a744-15b8c65b785a" (UID: "f3476fa7-d001-46e8-a744-15b8c65b785a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:24.629228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.629209 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-socket" pod "f3476fa7-d001-46e8-a744-15b8c65b785a" (UID: "f3476fa7-d001-46e8-a744-15b8c65b785a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.629446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.629429 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-certs" pod "f3476fa7-d001-46e8-a744-15b8c65b785a" (UID: "f3476fa7-d001-46e8-a744-15b8c65b785a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.629616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.629590 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-credential-socket" pod "f3476fa7-d001-46e8-a744-15b8c65b785a" (UID: "f3476fa7-d001-46e8-a744-15b8c65b785a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.630286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.630243 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-token" pod "f3476fa7-d001-46e8-a744-15b8c65b785a" (UID: "f3476fa7-d001-46e8-a744-15b8c65b785a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:24.630507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.630474 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-kube-api-access-76zjf" pod "f3476fa7-d001-46e8-a744-15b8c65b785a" (UID: "f3476fa7-d001-46e8-a744-15b8c65b785a"). InnerVolumeSpecName "kube-api-access-76zjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:24.631492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.631326 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-envoy" pod "f3476fa7-d001-46e8-a744-15b8c65b785a" (UID: "f3476fa7-d001-46e8-a744-15b8c65b785a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:24.631492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.631440 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-podinfo" pod "f3476fa7-d001-46e8-a744-15b8c65b785a" (UID: "f3476fa7-d001-46e8-a744-15b8c65b785a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:09:24.675464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.675425 5515 generic.go:356] "Generic (PLEG): container finished" podID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerID="76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789" exitCode=0 Oct 30 13:09:24.675706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.675487 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" Oct 30 13:09:24.675706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.675507 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" event={"ID":"f3476fa7-d001-46e8-a744-15b8c65b785a","Type":"ContainerDied","Data":"76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789"} Oct 30 13:09:24.675706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.675577 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc" event={"ID":"f3476fa7-d001-46e8-a744-15b8c65b785a","Type":"ContainerDied","Data":"074d609519b1008a458d6fbb488fcb98425259770588382994baac6df8df4ca1"} Oct 30 13:09:24.675706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.675610 5515 scope.go:122] "RemoveContainer" containerID="76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789" Oct 30 13:09:24.687857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.687836 5515 scope.go:122] "RemoveContainer" containerID="12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9" Oct 30 13:09:24.728603 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.728575 5515 scope.go:122] "RemoveContainer" containerID="76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789" Oct 30 13:09:24.728963 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:09:24.728937 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789\": container with ID starting with 76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789 not found: ID does not exist" containerID="76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789" Oct 30 13:09:24.729042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.728978 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789"} err="failed to get container status \"76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789\": rpc error: code = NotFound desc = could not find container \"76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789\": container with ID starting with 76cbe9771c0c9913e4f135a87c98f363c4df8bc5e998acf8beab87920747f789 not found: ID does not exist" Oct 30 13:09:24.729042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729006 5515 scope.go:122] "RemoveContainer" containerID="12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9" Oct 30 13:09:24.729655 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:09:24.729630 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9\": container with ID starting with 12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9 not found: ID does not exist" containerID="12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9" Oct 30 13:09:24.729718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729662 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9"} err="failed to get container status \"12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9\": rpc error: code = NotFound desc = could not find container \"12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9\": container with ID starting with 12d1058baa58ec167a1b6f607ad006f3ebb6f795d4630f603f1b0216f9f12ba9 not found: ID does not exist" Oct 30 13:09:24.729767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729731 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.729767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729754 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.729844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729768 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.729844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729783 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3476fa7-d001-46e8-a744-15b8c65b785a-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.729844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729796 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.729844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729809 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.729844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729820 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.729844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729832 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76zjf\" (UniqueName: \"kubernetes.io/projected/f3476fa7-d001-46e8-a744-15b8c65b785a-kube-api-access-76zjf\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.730046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.729857 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3476fa7-d001-46e8-a744-15b8c65b785a-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:09:24.744424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.744386 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc"] Oct 30 13:09:24.748238 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.748203 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-aa466d-openshift-default-5798999957-7gtpc"] Oct 30 13:09:24.912292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:24.909353 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:26.026703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:26.026662 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" path="/var/lib/kubelet/pods/f3476fa7-d001-46e8-a744-15b8c65b785a/volumes" Oct 30 13:09:26.522887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:26.522102 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:27.817309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:27.817207 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:28.266221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:28.266188 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:28.467037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:28.467008 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:28.670249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:28.670220 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:09:28.923964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:28.923879 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:28.989550 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:09:28.989513 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:09:29.014096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:29.014031 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bvptt_7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca/payload-processing/0.log" Oct 30 13:09:29.238534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:29.238453 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bvptt_7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca/payload-processing/0.log" Oct 30 13:09:29.436613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:29.436585 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf_3923d62a-9587-4c06-a628-2a7f012e0540/payload-processing/0.log" Oct 30 13:09:29.628732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:29.628651 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf_3923d62a-9587-4c06-a628-2a7f012e0540/payload-processing/0.log" Oct 30 13:09:29.986887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:29.986848 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:31.916162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:31.916123 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:33.771902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:33.771820 5515 generic.go:356] "Generic (PLEG): container finished" podID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerID="d249bd733c6ff50d0d1efb871a47e5016a5190ad126e5bf77522d751711b5505" exitCode=0 Oct 30 13:09:33.771902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:33.771891 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" event={"ID":"37fddc62-e822-4b6f-be6f-870ac7691dac","Type":"ContainerDied","Data":"d249bd733c6ff50d0d1efb871a47e5016a5190ad126e5bf77522d751711b5505"} Oct 30 13:09:33.772421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:33.772401 5515 scope.go:122] "RemoveContainer" containerID="d249bd733c6ff50d0d1efb871a47e5016a5190ad126e5bf77522d751711b5505" Oct 30 13:09:33.773937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:33.773907 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:33.774011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:33.773978 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:33.774049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:33.774017 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:09:34.277996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:34.277797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:34.782755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:34.782722 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" event={"ID":"37fddc62-e822-4b6f-be6f-870ac7691dac","Type":"ContainerStarted","Data":"805b42a70d0a8526324b027c80f12582c6674eb13a078339a2408ba13de84b76"} Oct 30 13:09:35.080333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:35.080246 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:09:35.080488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:35.080445 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:09:35.080530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:35.080483 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 13:09:36.080536 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:36.080498 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:09:36.080962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:36.080552 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 13:09:36.332953 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:36.332847 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:36.547296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:36.542355 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:36.787293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:36.767221 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:37.080539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:37.080464 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:09:37.080539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:37.080514 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 13:09:37.386503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:37.386458 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:37.606301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:37.602617 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:38.079542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:38.079501 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:09:38.079752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:38.079614 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 13:09:38.188327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:38.182195 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:39.080204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:39.080165 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:09:39.080427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:39.080237 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 13:09:40.014293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:40.004852 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:40.079861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:40.079822 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 13:09:40.080084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:40.079889 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 13:09:41.084452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:41.084414 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:09:41.084949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:41.084845 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:09:41.085734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:41.085716 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:09:41.970430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:41.970393 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:43.669314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:43.669253 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:44.906844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:44.906802 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:45.508107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:45.508062 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:46.164313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:46.164224 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:46.648798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:46.648766 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:46.860360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:46.857347 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:47.381782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:47.381740 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:48.977854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:48.977813 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:49.202020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:49.201988 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:49.718312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:49.718250 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:09:50.604198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:50.604162 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:50.816554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:50.816510 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:51.024678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:51.024644 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:52.718837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:52.718797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:52.963877 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:52.963827 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:53.292200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:53.292171 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:55.439984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:55.439947 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:56.972492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:56.972437 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:58.492451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:58.492409 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:09:58.492975 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:58.492482 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:58.579837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:09:58.576065 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:58.989643 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:09:58.989598 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:10:00.208681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:00.208646 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:01.077625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:01.077533 5515 generic.go:356] "Generic (PLEG): container finished" podID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerID="0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500" exitCode=0 Oct 30 13:10:01.077625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:01.077608 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" event={"ID":"c08af005-2812-4d6c-97d0-c7aea99250bd","Type":"ContainerDied","Data":"0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500"} Oct 30 13:10:01.078150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:01.078135 5515 scope.go:122] "RemoveContainer" containerID="0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500" Oct 30 13:10:01.079802 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:01.079772 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:01.079895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:01.079845 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:01.079895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:01.079877 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:01.927011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:01.926955 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:02.087712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:02.087677 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" event={"ID":"c08af005-2812-4d6c-97d0-c7aea99250bd","Type":"ContainerStarted","Data":"9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820"} Oct 30 13:10:02.460122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:02.460081 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:10:02.460379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:02.460336 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:10:02.460467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:02.460404 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:10:02.779054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:02.778953 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:03.291730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:03.291694 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:03.463291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:03.462675 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:10:03.463291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:03.462745 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:10:03.849543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:03.849500 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:04.046374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:04.046341 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:04.460451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:04.460414 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:10:04.460882 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:04.460482 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:10:04.537833 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:04.537797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:05.460487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:05.460452 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:10:05.460934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:05.460530 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:10:06.100851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:06.100808 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:06.461208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:06.461172 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 13:10:06.461656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:06.461244 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 13:10:07.464928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:07.464896 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:10:07.465490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:07.465467 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:10:07.466355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:07.466337 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:10:07.822763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:07.822659 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:08.031572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.031538 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:08.275029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.274999 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:08.545979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.545894 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d"] Oct 30 13:10:08.546588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.546570 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" Oct 30 13:10:08.546588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.546587 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" Oct 30 13:10:08.546739 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.546619 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" Oct 30 13:10:08.546739 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.546625 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" Oct 30 13:10:08.546739 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.546723 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" Oct 30 13:10:08.550638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.550619 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.582137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.582109 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d"] Oct 30 13:10:08.654854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.654818 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.655046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.654915 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.655046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.654996 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.655046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.655026 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.655217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.655174 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9vbc\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-kube-api-access-v9vbc\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.655217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.655205 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.655373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.655358 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.655441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.655393 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.655482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.655454 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.798882 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.798786 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vbc\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-kube-api-access-v9vbc\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.799361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.799313 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.799835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.799803 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.799988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.799897 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.800078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.800003 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.800138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.800102 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.800317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.800297 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.800493 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.800467 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.802869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.802842 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.803466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.803439 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.803782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.803757 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.804411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.804389 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.807894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.807865 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.808350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.808220 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.808484 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.808398 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.809607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.809585 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.828437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.828401 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.828602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.828539 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9vbc\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-kube-api-access-v9vbc\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:08.864148 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:08.864108 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:09.186341 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:10:09.186311 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab051748_23bd_4b4d_8b7a_dab99c1b8ec2.slice/crio-9ad9b4f7d5c1df1dce91722d34b7dc16734eb3502d1b7feaabc2dce9912540a6 WatchSource:0}: Error finding container 9ad9b4f7d5c1df1dce91722d34b7dc16734eb3502d1b7feaabc2dce9912540a6: Status 404 returned error can't find the container with id 9ad9b4f7d5c1df1dce91722d34b7dc16734eb3502d1b7feaabc2dce9912540a6 Oct 30 13:10:09.186582 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:09.186554 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d"] Oct 30 13:10:09.189348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:09.189311 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:09.189442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:09.189382 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:09.189442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:09.189412 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:09.493350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:09.487800 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:10.165572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.165527 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" event={"ID":"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2","Type":"ContainerStarted","Data":"ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893"} Oct 30 13:10:10.165996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.165604 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" event={"ID":"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2","Type":"ContainerStarted","Data":"9ad9b4f7d5c1df1dce91722d34b7dc16734eb3502d1b7feaabc2dce9912540a6"} Oct 30 13:10:10.247455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.247391 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podStartSLOduration=2.24737339 podStartE2EDuration="2.24737339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:10.233711931 +0000 UTC m=+2228.360512836" watchObservedRunningTime="2026-10-30 13:10:10.24737339 +0000 UTC m=+2228.374174288" Oct 30 13:10:10.721438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.721396 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg"] Oct 30 13:10:10.721924 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.721890 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" podUID="3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" containerName="maas-api" containerID="cri-o://6797a2b901ce0bdec1657c31068e5bd337b8b514e18ee6a0bcde139cfda8a050" gracePeriod=30 Oct 30 13:10:10.809443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.809406 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:10.864431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.864401 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:10.866183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.866156 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:10:10.866348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.866208 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:10:10.934295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.931661 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k"] Oct 30 13:10:10.934295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:10.932000 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" podUID="86d99a3b-3579-431d-8bd8-958afee0a9a0" containerName="payload-pre-processing" containerID="cri-o://9b96e0f99eae9ae9ff7c66f178adfd0ae9e6edba3192b29ee47ab9df7894cd77" gracePeriod=30 Oct 30 13:10:11.020011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.019825 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:11.044522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.044478 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" podUID="3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.123:8443/health\": dial tcp 10.134.0.123:8443: connect: connection refused" Oct 30 13:10:11.178454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.175019 5515 generic.go:356] "Generic (PLEG): container finished" podID="86d99a3b-3579-431d-8bd8-958afee0a9a0" containerID="9b96e0f99eae9ae9ff7c66f178adfd0ae9e6edba3192b29ee47ab9df7894cd77" exitCode=0 Oct 30 13:10:11.178454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.175166 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" event={"ID":"86d99a3b-3579-431d-8bd8-958afee0a9a0","Type":"ContainerDied","Data":"9b96e0f99eae9ae9ff7c66f178adfd0ae9e6edba3192b29ee47ab9df7894cd77"} Oct 30 13:10:11.178454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.177516 5515 generic.go:356] "Generic (PLEG): container finished" podID="3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" containerID="6797a2b901ce0bdec1657c31068e5bd337b8b514e18ee6a0bcde139cfda8a050" exitCode=0 Oct 30 13:10:11.179041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.178603 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" event={"ID":"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e","Type":"ContainerDied","Data":"6797a2b901ce0bdec1657c31068e5bd337b8b514e18ee6a0bcde139cfda8a050"} Oct 30 13:10:11.268366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.259750 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:11.481339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.481308 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:11.577348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.573618 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:10:11.588539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.588505 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:10:11.678737 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.678699 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-maas-api-tls\") pod \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " Oct 30 13:10:11.678930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.678766 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-metrics-certs\") pod \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " Oct 30 13:10:11.678930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.678805 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d99a3b-3579-431d-8bd8-958afee0a9a0-kube-api-access-89m9l\" (UniqueName: \"kubernetes.io/projected/86d99a3b-3579-431d-8bd8-958afee0a9a0-kube-api-access-89m9l\") pod \"86d99a3b-3579-431d-8bd8-958afee0a9a0\" (UID: \"86d99a3b-3579-431d-8bd8-958afee0a9a0\") " Oct 30 13:10:11.678930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.678911 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d99a3b-3579-431d-8bd8-958afee0a9a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d99a3b-3579-431d-8bd8-958afee0a9a0-plugins-config-volume\") pod \"86d99a3b-3579-431d-8bd8-958afee0a9a0\" (UID: \"86d99a3b-3579-431d-8bd8-958afee0a9a0\") " Oct 30 13:10:11.679068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.679004 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-kube-api-access-6fz22\" (UniqueName: \"kubernetes.io/projected/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-kube-api-access-6fz22\") pod \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " Oct 30 13:10:11.679239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.679215 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-openshift-service-ca\") pod \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\" (UID: \"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e\") " Oct 30 13:10:11.679398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.679373 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d99a3b-3579-431d-8bd8-958afee0a9a0-plugins-config-volume" pod "86d99a3b-3579-431d-8bd8-958afee0a9a0" (UID: "86d99a3b-3579-431d-8bd8-958afee0a9a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:11.679671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.679644 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-openshift-service-ca" pod "3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" (UID: "3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:11.680042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.680010 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d99a3b-3579-431d-8bd8-958afee0a9a0-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.680130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.680046 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.681182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.681157 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-maas-api-tls" pod "3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" (UID: "3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:11.681288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.681215 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d99a3b-3579-431d-8bd8-958afee0a9a0-kube-api-access-89m9l" pod "86d99a3b-3579-431d-8bd8-958afee0a9a0" (UID: "86d99a3b-3579-431d-8bd8-958afee0a9a0"). InnerVolumeSpecName "kube-api-access-89m9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:11.681288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.681235 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-kube-api-access-6fz22" pod "3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" (UID: "3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e"). InnerVolumeSpecName "kube-api-access-6fz22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:11.682540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.682518 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-metrics-certs" pod "3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" (UID: "3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:11.752713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.752675 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:11.798357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.794626 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.798357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.794669 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.798357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.794686 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89m9l\" (UniqueName: \"kubernetes.io/projected/86d99a3b-3579-431d-8bd8-958afee0a9a0-kube-api-access-89m9l\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.798357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.794858 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fz22\" (UniqueName: \"kubernetes.io/projected/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e-kube-api-access-6fz22\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.879300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.878233 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:10:11.879300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.878339 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:10:11.987322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:11.986986 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:12.184668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.184609 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9"] Oct 30 13:10:12.185215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.184868 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" containerID="cri-o://642a7cb45a8c2527191554489c677c5871a4d78f7e109e69427dc252f5ec3f25" gracePeriod=30 Oct 30 13:10:12.190516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.190489 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" Oct 30 13:10:12.190682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.190533 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k" event={"ID":"86d99a3b-3579-431d-8bd8-958afee0a9a0","Type":"ContainerDied","Data":"92977de93d76b3bc7a6bbb39babf498b91d0de8c3dc3d0ba77904d36bd5933bd"} Oct 30 13:10:12.190682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.190597 5515 scope.go:122] "RemoveContainer" containerID="9b96e0f99eae9ae9ff7c66f178adfd0ae9e6edba3192b29ee47ab9df7894cd77" Oct 30 13:10:12.193133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.193106 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" Oct 30 13:10:12.193338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.193152 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg" event={"ID":"3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e","Type":"ContainerDied","Data":"9fc5a8afcde255ef041d1218f3c3965061656d50e6b223f06367aaec05f0f128"} Oct 30 13:10:12.227869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.226928 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:12.237075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.234912 5515 scope.go:122] "RemoveContainer" containerID="6797a2b901ce0bdec1657c31068e5bd337b8b514e18ee6a0bcde139cfda8a050" Oct 30 13:10:12.341784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.341593 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg"] Oct 30 13:10:12.416332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.405606 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9c553-f5c66588-7lvvg"] Oct 30 13:10:12.416332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.412094 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k"] Oct 30 13:10:12.416332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.415616 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9c553-959488f9f-nvj4k"] Oct 30 13:10:12.440290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.436705 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:12.864440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.864410 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:10:12.864619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:12.864462 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:10:13.041234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.041188 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5"] Oct 30 13:10:13.041856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.041841 5515 cpu_manager.go:414] "containerMap: removing container" podUID="86d99a3b-3579-431d-8bd8-958afee0a9a0" containerName="payload-pre-processing" Oct 30 13:10:13.041899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.041858 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d99a3b-3579-431d-8bd8-958afee0a9a0" containerName="payload-pre-processing" Oct 30 13:10:13.041931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.041903 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" containerName="maas-api" Oct 30 13:10:13.041931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.041909 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" containerName="maas-api" Oct 30 13:10:13.042006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.041997 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3476fa7-d001-46e8-a744-15b8c65b785a" containerName="istio-proxy" Oct 30 13:10:13.042047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.042010 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" containerName="maas-api" Oct 30 13:10:13.042047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.042018 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d99a3b-3579-431d-8bd8-958afee0a9a0" containerName="payload-pre-processing" Oct 30 13:10:13.045459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.045434 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.063483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.063422 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6"] Oct 30 13:10:13.066973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.066946 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b"] Oct 30 13:10:13.067123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.067061 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.072981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.071245 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.072981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.072783 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb86fd-openshift-default-dockercfg-bqgfs\"" Oct 30 13:10:13.077031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.077006 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5"] Oct 30 13:10:13.116342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.114628 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6"] Oct 30 13:10:13.116342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.114721 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b"] Oct 30 13:10:13.153872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.153840 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.154034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.153896 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.154034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.153954 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.154192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154094 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.154192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154150 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.154192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154178 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.154388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154302 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.154388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154331 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6nmx\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-kube-api-access-m6nmx\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.154388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154368 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.154552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154395 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.154552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154411 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.154552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154431 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshpz\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-kube-api-access-xshpz\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.154552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154480 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.154741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154548 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.154741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154621 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.154741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154699 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.154741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154721 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.154943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154750 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.154943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154766 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.154943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154788 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.154943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154908 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.154943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154935 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.155176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154957 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhpj\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-kube-api-access-vdhpj\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.155176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.154987 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.155176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.155007 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.155176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.155033 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.155176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.155061 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42defd-792a-4658-8ac3-e7fef9e4d459-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.256039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.255989 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256050 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256104 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256131 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshpz\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-kube-api-access-xshpz\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256170 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256203 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256237 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256299 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256337 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256388 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256412 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256447 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256535 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256539 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256559 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256572 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.256584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256585 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256618 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256622 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhpj\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-kube-api-access-vdhpj\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256700 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256747 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256798 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256842 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256888 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256848 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42defd-792a-4658-8ac3-e7fef9e4d459-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256972 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.256984 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257046 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257077 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257112 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257167 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257178 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257198 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257244 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257285 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6nmx\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-kube-api-access-m6nmx\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257373 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.257468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257113 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.258971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257640 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.258971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.257982 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.258971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.258077 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42defd-792a-4658-8ac3-e7fef9e4d459-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.258971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.258103 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.258971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.258104 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.259517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.259495 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.259853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.259831 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.259920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.259904 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.260097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.260078 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.260417 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.260401 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.260639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.260616 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.283521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.283487 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshpz\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-kube-api-access-xshpz\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.285732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.285699 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.285880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.285737 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhpj\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-kube-api-access-vdhpj\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.287457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.287433 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6nmx\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-kube-api-access-m6nmx\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.287741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.287720 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.287830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.287813 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.363451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.363415 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:13.392079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.392047 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:13.431767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.431735 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:13.722810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.722779 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5"] Oct 30 13:10:13.725322 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:10:13.725297 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac38c16d_786e_42c2_ae5e_42c2a2bef0da.slice/crio-553db0fc1604dafa6129a9f91f14985a6c8f8ce9decf4e25551561891c9f8b99 WatchSource:0}: Error finding container 553db0fc1604dafa6129a9f91f14985a6c8f8ce9decf4e25551561891c9f8b99: Status 404 returned error can't find the container with id 553db0fc1604dafa6129a9f91f14985a6c8f8ce9decf4e25551561891c9f8b99 Oct 30 13:10:13.728374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.728343 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:13.728464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.728411 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:13.728464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.728445 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:13.779165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.779139 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6"] Oct 30 13:10:13.780981 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:10:13.780948 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47806efb_6a33_4d7b_8b2f_a0eb42dcb169.slice/crio-3332a55796e7b9d91b85ff7904f43617ec75c91eb8f5042bb08e2b18a28c3255 WatchSource:0}: Error finding container 3332a55796e7b9d91b85ff7904f43617ec75c91eb8f5042bb08e2b18a28c3255: Status 404 returned error can't find the container with id 3332a55796e7b9d91b85ff7904f43617ec75c91eb8f5042bb08e2b18a28c3255 Oct 30 13:10:13.783542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.783508 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:13.783634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.783575 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:13.783634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.783602 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:13.827397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.823302 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b"] Oct 30 13:10:13.827685 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:10:13.827656 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff42defd_792a_4658_8ac3_e7fef9e4d459.slice/crio-2147d56f17e940679015182d6f82fb809a536468ec29fabfd5df1482225197cb WatchSource:0}: Error finding container 2147d56f17e940679015182d6f82fb809a536468ec29fabfd5df1482225197cb: Status 404 returned error can't find the container with id 2147d56f17e940679015182d6f82fb809a536468ec29fabfd5df1482225197cb Oct 30 13:10:13.830997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.830965 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:13.831105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.831035 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:13.831105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.831065 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:10:13.869664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.869627 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:10:13.869821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:13.869691 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:10:14.044721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.044610 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e" path="/var/lib/kubelet/pods/3b6dfb2a-cf2e-4bc0-8df3-2d92307afa9e/volumes" Oct 30 13:10:14.045669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.045644 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d99a3b-3579-431d-8bd8-958afee0a9a0" path="/var/lib/kubelet/pods/86d99a3b-3579-431d-8bd8-958afee0a9a0/volumes" Oct 30 13:10:14.243574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.243536 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" event={"ID":"ff42defd-792a-4658-8ac3-e7fef9e4d459","Type":"ContainerStarted","Data":"044ead6e5ce305a04a0f95a5e9c0f188b34cc8969098fcd4582354a64086999d"} Oct 30 13:10:14.243839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.243822 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" event={"ID":"ff42defd-792a-4658-8ac3-e7fef9e4d459","Type":"ContainerStarted","Data":"2147d56f17e940679015182d6f82fb809a536468ec29fabfd5df1482225197cb"} Oct 30 13:10:14.245674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.245644 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" event={"ID":"47806efb-6a33-4d7b-8b2f-a0eb42dcb169","Type":"ContainerStarted","Data":"bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664"} Oct 30 13:10:14.245813 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.245688 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" event={"ID":"47806efb-6a33-4d7b-8b2f-a0eb42dcb169","Type":"ContainerStarted","Data":"3332a55796e7b9d91b85ff7904f43617ec75c91eb8f5042bb08e2b18a28c3255"} Oct 30 13:10:14.249285 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.249025 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" event={"ID":"ac38c16d-786e-42c2-ae5e-42c2a2bef0da","Type":"ContainerStarted","Data":"6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf"} Oct 30 13:10:14.249285 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.249067 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" event={"ID":"ac38c16d-786e-42c2-ae5e-42c2a2bef0da","Type":"ContainerStarted","Data":"553db0fc1604dafa6129a9f91f14985a6c8f8ce9decf4e25551561891c9f8b99"} Oct 30 13:10:14.338366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.294677 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podStartSLOduration=1.29465708 podStartE2EDuration="1.29465708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:14.293593207 +0000 UTC m=+2232.420394108" watchObservedRunningTime="2026-10-30 13:10:14.29465708 +0000 UTC m=+2232.421457980" Oct 30 13:10:14.395805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.384414 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:14.395805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.385520 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:14.436367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.432882 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:14.440047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.437729 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:14.440047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.437787 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:14.440047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.438035 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:10:14.440047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.438070 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 13:10:14.440569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.440473 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:14.440864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.440778 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:14.440864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.440816 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:14.453961 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.453895 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podStartSLOduration=1.453870144 podStartE2EDuration="1.453870144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:14.383779891 +0000 UTC m=+2232.510580790" watchObservedRunningTime="2026-10-30 13:10:14.453870144 +0000 UTC m=+2232.580671043" Oct 30 13:10:14.474546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.473846 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podStartSLOduration=1.473825274 podStartE2EDuration="1.473825274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:14.47057267 +0000 UTC m=+2232.597373581" watchObservedRunningTime="2026-10-30 13:10:14.473825274 +0000 UTC m=+2232.600626174" Oct 30 13:10:14.867085 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.867044 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:10:14.867413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:14.867379 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:10:15.371214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:15.370742 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:15.371214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:15.370812 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:15.392482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:15.392448 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:10:15.392766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:15.392740 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 13:10:15.433555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:15.433519 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:15.433882 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:15.433805 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:15.925597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:15.925070 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:10:15.925597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:15.925144 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:10:16.236474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:16.236368 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:16.378328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:16.375472 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:16.378328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:16.375532 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:16.460295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:16.459875 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:10:16.460295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:16.459982 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 13:10:16.477301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:16.461718 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:16.477301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:16.461789 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:16.867763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:16.867719 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:10:16.868109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:16.868074 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:10:17.317104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.314384 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:17.384293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.378483 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:17.384293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.378563 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:17.404820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.404782 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:10:17.405131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.405086 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 13:10:17.441893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.441855 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:17.442183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.442152 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:17.489353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.488434 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:17.536972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.536934 5515 generic.go:356] "Generic (PLEG): container finished" podID="ec9fce36-454e-4192-92ba-04acf3135f01" containerID="642a7cb45a8c2527191554489c677c5871a4d78f7e109e69427dc252f5ec3f25" exitCode=0 Oct 30 13:10:17.537281 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.537242 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" event={"ID":"ec9fce36-454e-4192-92ba-04acf3135f01","Type":"ContainerDied","Data":"642a7cb45a8c2527191554489c677c5871a4d78f7e109e69427dc252f5ec3f25"} Oct 30 13:10:17.537484 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.537447 5515 scope.go:122] "RemoveContainer" containerID="c966beb04a976c2a607ef367ef963cfc25b564c66ec09dd6305637d6ffff42ba" Oct 30 13:10:17.539309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.539283 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:17.937301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.921516 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:17.992384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:17.955045 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 13:10:18.256506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.231705 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:18.538364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.521449 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:18.538364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.521533 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:18.565303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.544922 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:18.565303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.558944 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 13:10:18.587294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.583687 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:10:18.650681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.648524 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 13:10:18.654293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.652772 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:10:18.654293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.652845 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 13:10:18.730293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.698497 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:18.730293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.698579 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:18.768301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.748476 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:18.847843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.847747 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-socket\") pod \"ec9fce36-454e-4192-92ba-04acf3135f01\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " Oct 30 13:10:18.848193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.848169 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9fce36-454e-4192-92ba-04acf3135f01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fce36-454e-4192-92ba-04acf3135f01-istiod-ca-cert\") pod \"ec9fce36-454e-4192-92ba-04acf3135f01\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " Oct 30 13:10:18.856291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.848369 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-istio-token\") pod \"ec9fce36-454e-4192-92ba-04acf3135f01\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " Oct 30 13:10:18.856597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.856557 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-data\") pod \"ec9fce36-454e-4192-92ba-04acf3135f01\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " Oct 30 13:10:18.861293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.858049 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-kube-api-access-fzx5p\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-kube-api-access-fzx5p\") pod \"ec9fce36-454e-4192-92ba-04acf3135f01\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " Oct 30 13:10:18.861866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.861589 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-envoy\") pod \"ec9fce36-454e-4192-92ba-04acf3135f01\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " Oct 30 13:10:18.862099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.862080 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec9fce36-454e-4192-92ba-04acf3135f01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fce36-454e-4192-92ba-04acf3135f01-istio-podinfo\") pod \"ec9fce36-454e-4192-92ba-04acf3135f01\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " Oct 30 13:10:18.862350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.862330 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-credential-socket\") pod \"ec9fce36-454e-4192-92ba-04acf3135f01\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " Oct 30 13:10:18.866294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.862795 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-certs\") pod \"ec9fce36-454e-4192-92ba-04acf3135f01\" (UID: \"ec9fce36-454e-4192-92ba-04acf3135f01\") " Oct 30 13:10:18.902692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:18.902650 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-data" pod "ec9fce36-454e-4192-92ba-04acf3135f01" (UID: "ec9fce36-454e-4192-92ba-04acf3135f01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:19.023420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.010921 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9fce36-454e-4192-92ba-04acf3135f01-istiod-ca-cert" pod "ec9fce36-454e-4192-92ba-04acf3135f01" (UID: "ec9fce36-454e-4192-92ba-04acf3135f01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:19.023420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.011214 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-istio-token" pod "ec9fce36-454e-4192-92ba-04acf3135f01" (UID: "ec9fce36-454e-4192-92ba-04acf3135f01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:19.023420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.021183 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 13:10:19.090320 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.077341 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:19.090320 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.077433 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:19.156499 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.156461 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-socket" pod "ec9fce36-454e-4192-92ba-04acf3135f01" (UID: "ec9fce36-454e-4192-92ba-04acf3135f01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:19.184291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.164801 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-credential-socket" pod "ec9fce36-454e-4192-92ba-04acf3135f01" (UID: "ec9fce36-454e-4192-92ba-04acf3135f01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:19.184291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.182931 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-kube-api-access-fzx5p" pod "ec9fce36-454e-4192-92ba-04acf3135f01" (UID: "ec9fce36-454e-4192-92ba-04acf3135f01"). InnerVolumeSpecName "kube-api-access-fzx5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:19.251304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.247758 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.251304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.247974 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.277296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.274777 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fce36-454e-4192-92ba-04acf3135f01-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.277296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.274816 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.277296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.274831 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.277296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.274846 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzx5p\" (UniqueName: \"kubernetes.io/projected/ec9fce36-454e-4192-92ba-04acf3135f01-kube-api-access-fzx5p\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.281292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.279372 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-certs" pod "ec9fce36-454e-4192-92ba-04acf3135f01" (UID: "ec9fce36-454e-4192-92ba-04acf3135f01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:19.314076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.314037 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:19.315147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.315099 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:19.379422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.372124 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:19.380191 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.380152 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:19.380375 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:10:19.375144 5515 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.108s" Oct 30 13:10:19.380592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.380579 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 13:10:19.380742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.380730 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 13:10:19.380874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.380864 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 13:10:19.380995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.380986 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 13:10:19.381969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.374492 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec9fce36-454e-4192-92ba-04acf3135f01-istio-podinfo" pod "ec9fce36-454e-4192-92ba-04acf3135f01" (UID: "ec9fce36-454e-4192-92ba-04acf3135f01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:10:19.394399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.394363 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:10:19.394652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.394626 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 13:10:19.407684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.407589 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fce36-454e-4192-92ba-04acf3135f01-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.412347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.412314 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.431565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.431527 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" event={"ID":"ec9fce36-454e-4192-92ba-04acf3135f01","Type":"ContainerDied","Data":"6337cadb9b105015789122bae0a491812ef5049ece86df263e06b2d19b8c4798"} Oct 30 13:10:19.526294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.517386 5515 scope.go:122] "RemoveContainer" containerID="642a7cb45a8c2527191554489c677c5871a4d78f7e109e69427dc252f5ec3f25" Oct 30 13:10:19.526294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.432011 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9" Oct 30 13:10:19.526294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.432456 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:19.526294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.517821 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:19.526294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.476403 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-envoy" pod "ec9fce36-454e-4192-92ba-04acf3135f01" (UID: "ec9fce36-454e-4192-92ba-04acf3135f01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:19.534312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.530809 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:19.599294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.576624 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fce36-454e-4192-92ba-04acf3135f01-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:19.687382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.660790 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:10:19.687382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.680376 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:19.687382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.680486 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 13:10:19.687382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.681656 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:19.687382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.684193 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:19.690617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.690583 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:10:19.759848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.759788 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:19.760756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.760719 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 13:10:19.947566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.947535 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 13:10:19.986613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:19.986585 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 13:10:20.160717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.158389 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 13:10:20.349482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.349381 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:20.405294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.383092 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:20.478945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.478904 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:20.479175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.407344 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:10:20.506439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.506387 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 13:10:20.506698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.458261 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:20.506835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.506815 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:20.739292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.738470 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:20.799588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:20.798469 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:21.122349 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:10:21.089026 5515 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.093s" Oct 30 13:10:21.124178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.124144 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9"] Oct 30 13:10:21.166719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.132836 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:21.199308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.168851 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:21.242323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.237831 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f9ea8f-openshift-default-545548f9cc-wvrl9"] Oct 30 13:10:21.377397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.376731 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Oct 30 13:10:21.377397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.377087 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:21.377397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.377124 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:21.396293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.392396 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:21.398137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.397066 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:10:21.398137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.397127 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 13:10:21.455490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.455435 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:21.455783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.455735 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:21.795505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.795462 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:21.796421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.796393 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:21.817294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.801457 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:21.904298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.899741 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 13:10:21.976528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.976401 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 13:10:21.987298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.987240 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 13:10:21.990768 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:21.990578 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:22.083680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:22.083577 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" path="/var/lib/kubelet/pods/ec9fce36-454e-4192-92ba-04acf3135f01/volumes" Oct 30 13:10:22.084114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:22.084092 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 13:10:22.364152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:22.364056 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:22.364435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:22.364387 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:22.393311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:22.393248 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 13:10:22.393528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:22.393334 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 13:10:22.433382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:22.433334 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:22.433605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:22.433407 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:22.786354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:22.786314 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:23.281137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:23.281098 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l"] Oct 30 13:10:23.367052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:23.366171 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 13:10:23.375872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:23.369389 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 13:10:23.434658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:23.434623 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 13:10:23.434900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:23.434870 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 13:10:23.577015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:23.566368 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:23.813285 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:23.813237 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:24.419131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:24.419080 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:24.419686 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:24.419154 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.581324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:24.580765 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:25.419292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:25.418372 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:25.419292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:25.418442 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.435671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:25.435263 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:25.435671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:25.435346 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.461412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:25.461359 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:25.461619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:25.461433 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.050039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.047533 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:26.050039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.048844 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:26.066795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.066755 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:10:26.192976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.192939 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:26.369843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.369747 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:26.370138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.370108 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:26.371132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.371110 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:10:26.476464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.476426 5515 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:26.477714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.477682 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.602830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.602800 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:26.603398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.603369 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:26.604414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:26.604388 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:10:27.816383 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:27.816351 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:28.282890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.282841 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" podUID="0bd6c2b6-791c-4133-9033-2ced2194d2a2" containerName="payload-processing" containerID="cri-o://1183d6d09f16ec9aec006a7baa348d2309f1fff98b4bb9c6c1a611e143d798b0" gracePeriod=25 Oct 30 13:10:28.419524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.419489 5515 generic.go:356] "Generic (PLEG): container finished" podID="0bd6c2b6-791c-4133-9033-2ced2194d2a2" containerID="1183d6d09f16ec9aec006a7baa348d2309f1fff98b4bb9c6c1a611e143d798b0" exitCode=0 Oct 30 13:10:28.419691 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.419568 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" event={"ID":"0bd6c2b6-791c-4133-9033-2ced2194d2a2","Type":"ContainerDied","Data":"1183d6d09f16ec9aec006a7baa348d2309f1fff98b4bb9c6c1a611e143d798b0"} Oct 30 13:10:28.565619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.565590 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:10:28.713988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.713951 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd6c2b6-791c-4133-9033-2ced2194d2a2-kube-api-access-vqdrn\" (UniqueName: \"kubernetes.io/projected/0bd6c2b6-791c-4133-9033-2ced2194d2a2-kube-api-access-vqdrn\") pod \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\" (UID: \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\") " Oct 30 13:10:28.714240 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.714224 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd6c2b6-791c-4133-9033-2ced2194d2a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd6c2b6-791c-4133-9033-2ced2194d2a2-plugins-config-volume\") pod \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\" (UID: \"0bd6c2b6-791c-4133-9033-2ced2194d2a2\") " Oct 30 13:10:28.714615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.714589 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd6c2b6-791c-4133-9033-2ced2194d2a2-plugins-config-volume" pod "0bd6c2b6-791c-4133-9033-2ced2194d2a2" (UID: "0bd6c2b6-791c-4133-9033-2ced2194d2a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:28.714761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.714745 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd6c2b6-791c-4133-9033-2ced2194d2a2-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:28.716321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.716297 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd6c2b6-791c-4133-9033-2ced2194d2a2-kube-api-access-vqdrn" pod "0bd6c2b6-791c-4133-9033-2ced2194d2a2" (UID: "0bd6c2b6-791c-4133-9033-2ced2194d2a2"). InnerVolumeSpecName "kube-api-access-vqdrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:28.816225 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.816147 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqdrn\" (UniqueName: \"kubernetes.io/projected/0bd6c2b6-791c-4133-9033-2ced2194d2a2-kube-api-access-vqdrn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:28.983776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.983734 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr"] Oct 30 13:10:28.984252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:28.983992 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" podUID="dd36fae0-af4b-46c6-ac38-99740e6bd1c7" containerName="maas-api" containerID="cri-o://3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76" gracePeriod=30 Oct 30 13:10:29.003569 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:10:29.003537 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:10:29.136960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.136922 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:29.359350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.355785 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:10:29.470494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.456774 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-maas-api-tls\") pod \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " Oct 30 13:10:29.470494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.456835 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-openshift-service-ca\") pod \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " Oct 30 13:10:29.470494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.457162 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-metrics-certs\") pod \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " Oct 30 13:10:29.470494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.457207 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-kube-api-access-n5rdz\" (UniqueName: \"kubernetes.io/projected/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-kube-api-access-n5rdz\") pod \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\" (UID: \"dd36fae0-af4b-46c6-ac38-99740e6bd1c7\") " Oct 30 13:10:29.470494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.470413 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-openshift-service-ca" pod "dd36fae0-af4b-46c6-ac38-99740e6bd1c7" (UID: "dd36fae0-af4b-46c6-ac38-99740e6bd1c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:29.525640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.510454 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-kube-api-access-n5rdz" pod "dd36fae0-af4b-46c6-ac38-99740e6bd1c7" (UID: "dd36fae0-af4b-46c6-ac38-99740e6bd1c7"). InnerVolumeSpecName "kube-api-access-n5rdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:29.525640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.514408 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-metrics-certs" pod "dd36fae0-af4b-46c6-ac38-99740e6bd1c7" (UID: "dd36fae0-af4b-46c6-ac38-99740e6bd1c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:29.529680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.528062 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" event={"ID":"0bd6c2b6-791c-4133-9033-2ced2194d2a2","Type":"ContainerDied","Data":"bc3d15f72e1c3aee198d061f5b72c4f4127893413fa2ff0f52b36603303ba1c9"} Oct 30 13:10:29.529680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.528123 5515 scope.go:122] "RemoveContainer" containerID="1183d6d09f16ec9aec006a7baa348d2309f1fff98b4bb9c6c1a611e143d798b0" Oct 30 13:10:29.529680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.528308 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l" Oct 30 13:10:29.610933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.538843 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-maas-api-tls" pod "dd36fae0-af4b-46c6-ac38-99740e6bd1c7" (UID: "dd36fae0-af4b-46c6-ac38-99740e6bd1c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:29.610933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.560532 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:29.610933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.560576 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rdz\" (UniqueName: \"kubernetes.io/projected/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-kube-api-access-n5rdz\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:29.610933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.560591 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:29.610933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.560606 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd36fae0-af4b-46c6-ac38-99740e6bd1c7-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:29.632931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.613617 5515 generic.go:356] "Generic (PLEG): container finished" podID="dd36fae0-af4b-46c6-ac38-99740e6bd1c7" containerID="3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76" exitCode=0 Oct 30 13:10:29.632931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.613694 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" event={"ID":"dd36fae0-af4b-46c6-ac38-99740e6bd1c7","Type":"ContainerDied","Data":"3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76"} Oct 30 13:10:29.632931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.613754 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" event={"ID":"dd36fae0-af4b-46c6-ac38-99740e6bd1c7","Type":"ContainerDied","Data":"c75ab476add17df2e5b869d77a96ec71702773e260b97bbe315a67802a302f6b"} Oct 30 13:10:29.632931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.613900 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr" Oct 30 13:10:29.686433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.686407 5515 scope.go:122] "RemoveContainer" containerID="3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76" Oct 30 13:10:29.706297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.706233 5515 scope.go:122] "RemoveContainer" containerID="3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76" Oct 30 13:10:29.706786 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:10:29.706738 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76\": container with ID starting with 3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76 not found: ID does not exist" containerID="3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76" Oct 30 13:10:29.706922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.706797 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76"} err="failed to get container status \"3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76\": rpc error: code = NotFound desc = could not find container \"3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76\": container with ID starting with 3f898c451d7a5821a7b5ac55473655a4433a54240f94f2f350d31989b58aef76 not found: ID does not exist" Oct 30 13:10:29.717475 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.717440 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l"] Oct 30 13:10:29.721755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.721719 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d84f47-68f6ff8b56-hrf5l"] Oct 30 13:10:29.725043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.725012 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:29.759222 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.759184 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr"] Oct 30 13:10:29.762084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:29.762051 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d84f47-cf9d9564f-ldfwr"] Oct 30 13:10:30.002030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:30.001939 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd6c2b6-791c-4133-9033-2ced2194d2a2" path="/var/lib/kubelet/pods/0bd6c2b6-791c-4133-9033-2ced2194d2a2/volumes" Oct 30 13:10:30.002547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:30.002528 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd36fae0-af4b-46c6-ac38-99740e6bd1c7" path="/var/lib/kubelet/pods/dd36fae0-af4b-46c6-ac38-99740e6bd1c7/volumes" Oct 30 13:10:30.002955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:30.002933 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:30.473009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:30.472974 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:31.307170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:31.307130 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:32.347051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:32.347020 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:33.939952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:33.939898 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:35.557191 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:35.556601 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:37.384115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:37.384079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:38.205002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:38.204890 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:38.422953 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:38.420501 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:38.437798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:38.437761 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk"] Oct 30 13:10:38.438075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:38.438039 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" containerID="cri-o://805b42a70d0a8526324b027c80f12582c6674eb13a078339a2408ba13de84b76" gracePeriod=30 Oct 30 13:10:38.695290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:38.687523 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:38.999680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:38.999593 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:40.992085 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:40.986257 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:41.087442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:41.087400 5515 patch_prober.go:29] interesting pod/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:10:41.087637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:41.087505 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:10:42.749992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:42.749904 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:42.889839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:42.889804 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:10:43.625338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:43.619478 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt"] Oct 30 13:10:43.687309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:43.683959 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb"] Oct 30 13:10:43.931947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:43.931902 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:44.270909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.270814 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:44.276715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.276678 5515 generic.go:356] "Generic (PLEG): container finished" podID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerID="805b42a70d0a8526324b027c80f12582c6674eb13a078339a2408ba13de84b76" exitCode=0 Oct 30 13:10:44.283604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.283576 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:10:44.286722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.286688 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" event={"ID":"37fddc62-e822-4b6f-be6f-870ac7691dac","Type":"ContainerDied","Data":"805b42a70d0a8526324b027c80f12582c6674eb13a078339a2408ba13de84b76"} Oct 30 13:10:44.291407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.291378 5515 scope.go:122] "RemoveContainer" containerID="805b42a70d0a8526324b027c80f12582c6674eb13a078339a2408ba13de84b76" Oct 30 13:10:44.309259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.309151 5515 scope.go:122] "RemoveContainer" containerID="d249bd733c6ff50d0d1efb871a47e5016a5190ad126e5bf77522d751711b5505" Oct 30 13:10:44.384292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.376193 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-data\") pod \"37fddc62-e822-4b6f-be6f-870ac7691dac\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " Oct 30 13:10:44.384292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.376280 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-socket\") pod \"37fddc62-e822-4b6f-be6f-870ac7691dac\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " Oct 30 13:10:44.384292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.376364 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-token\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-token\") pod \"37fddc62-e822-4b6f-be6f-870ac7691dac\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " Oct 30 13:10:44.384292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.376428 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-envoy\") pod \"37fddc62-e822-4b6f-be6f-870ac7691dac\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " Oct 30 13:10:44.384292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.376465 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-credential-socket\") pod \"37fddc62-e822-4b6f-be6f-870ac7691dac\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " Oct 30 13:10:44.384292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.376578 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-podinfo\") pod \"37fddc62-e822-4b6f-be6f-870ac7691dac\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " Oct 30 13:10:44.384292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.376654 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-certs\") pod \"37fddc62-e822-4b6f-be6f-870ac7691dac\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " Oct 30 13:10:44.384292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.376751 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fddc62-e822-4b6f-be6f-870ac7691dac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fddc62-e822-4b6f-be6f-870ac7691dac-istiod-ca-cert\") pod \"37fddc62-e822-4b6f-be6f-870ac7691dac\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " Oct 30 13:10:44.384292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.376785 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-kube-api-access-n5dkb\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-kube-api-access-n5dkb\") pod \"37fddc62-e822-4b6f-be6f-870ac7691dac\" (UID: \"37fddc62-e822-4b6f-be6f-870ac7691dac\") " Oct 30 13:10:44.427631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.427256 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-token" pod "37fddc62-e822-4b6f-be6f-870ac7691dac" (UID: "37fddc62-e822-4b6f-be6f-870ac7691dac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:44.428310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.427962 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-socket" pod "37fddc62-e822-4b6f-be6f-870ac7691dac" (UID: "37fddc62-e822-4b6f-be6f-870ac7691dac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:44.431024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.430788 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-envoy" pod "37fddc62-e822-4b6f-be6f-870ac7691dac" (UID: "37fddc62-e822-4b6f-be6f-870ac7691dac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:44.431684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.431442 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fddc62-e822-4b6f-be6f-870ac7691dac-istiod-ca-cert" pod "37fddc62-e822-4b6f-be6f-870ac7691dac" (UID: "37fddc62-e822-4b6f-be6f-870ac7691dac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:44.432146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.431852 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-credential-socket" pod "37fddc62-e822-4b6f-be6f-870ac7691dac" (UID: "37fddc62-e822-4b6f-be6f-870ac7691dac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:44.435955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.434399 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-data" pod "37fddc62-e822-4b6f-be6f-870ac7691dac" (UID: "37fddc62-e822-4b6f-be6f-870ac7691dac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:44.436612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.436561 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-kube-api-access-n5dkb" pod "37fddc62-e822-4b6f-be6f-870ac7691dac" (UID: "37fddc62-e822-4b6f-be6f-870ac7691dac"). InnerVolumeSpecName "kube-api-access-n5dkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:44.442980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.442949 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-podinfo" pod "37fddc62-e822-4b6f-be6f-870ac7691dac" (UID: "37fddc62-e822-4b6f-be6f-870ac7691dac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:10:44.471407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.465391 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-certs" pod "37fddc62-e822-4b6f-be6f-870ac7691dac" (UID: "37fddc62-e822-4b6f-be6f-870ac7691dac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:44.503726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.502362 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fddc62-e822-4b6f-be6f-870ac7691dac-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:44.503726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.502397 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5dkb\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-kube-api-access-n5dkb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:44.503726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.502413 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:44.503726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.502426 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:44.503726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.502439 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:44.503726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.502452 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:44.503726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.502465 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:44.503726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.502477 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fddc62-e822-4b6f-be6f-870ac7691dac-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:44.503726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:44.502490 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fddc62-e822-4b6f-be6f-870ac7691dac-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:45.313566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:45.313530 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" Oct 30 13:10:45.345989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:45.343226 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk" event={"ID":"37fddc62-e822-4b6f-be6f-870ac7691dac","Type":"ContainerDied","Data":"427a6d97fc0211d00e9e317555abf5aca758598c6d869de0bc5419e6375896c1"} Oct 30 13:10:45.411294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:45.409451 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:45.444293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:45.441406 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk"] Oct 30 13:10:45.486298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:45.482011 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d84f47-openshift-default-76dcd5c7b5-st9zk"] Oct 30 13:10:45.831295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:45.828402 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:10:45.831295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:45.828508 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:46.113429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:46.111221 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" path="/var/lib/kubelet/pods/37fddc62-e822-4b6f-be6f-870ac7691dac/volumes" Oct 30 13:10:47.726764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:47.726725 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:48.620181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:48.620138 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" podUID="3f7eb4d7-8164-4074-be07-de089f528ca7" containerName="payload-processing" containerID="cri-o://222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22" gracePeriod=25 Oct 30 13:10:48.686174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:48.686124 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" podUID="ce00f683-b3db-4b8a-8734-61b877751893" containerName="payload-pre-processing" containerID="cri-o://a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92" gracePeriod=25 Oct 30 13:10:48.995183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:48.995151 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:10:49.018654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.018628 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:10:49.123422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.123391 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7eb4d7-8164-4074-be07-de089f528ca7-kube-api-access-t76tf\" (UniqueName: \"kubernetes.io/projected/3f7eb4d7-8164-4074-be07-de089f528ca7-kube-api-access-t76tf\") pod \"3f7eb4d7-8164-4074-be07-de089f528ca7\" (UID: \"3f7eb4d7-8164-4074-be07-de089f528ca7\") " Oct 30 13:10:49.123422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.123425 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce00f683-b3db-4b8a-8734-61b877751893-kube-api-access-nrknl\" (UniqueName: \"kubernetes.io/projected/ce00f683-b3db-4b8a-8734-61b877751893-kube-api-access-nrknl\") pod \"ce00f683-b3db-4b8a-8734-61b877751893\" (UID: \"ce00f683-b3db-4b8a-8734-61b877751893\") " Oct 30 13:10:49.123650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.123522 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7eb4d7-8164-4074-be07-de089f528ca7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eb4d7-8164-4074-be07-de089f528ca7-plugins-config-volume\") pod \"3f7eb4d7-8164-4074-be07-de089f528ca7\" (UID: \"3f7eb4d7-8164-4074-be07-de089f528ca7\") " Oct 30 13:10:49.123650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.123588 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce00f683-b3db-4b8a-8734-61b877751893-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce00f683-b3db-4b8a-8734-61b877751893-plugins-config-volume\") pod \"ce00f683-b3db-4b8a-8734-61b877751893\" (UID: \"ce00f683-b3db-4b8a-8734-61b877751893\") " Oct 30 13:10:49.123997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.123969 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7eb4d7-8164-4074-be07-de089f528ca7-plugins-config-volume" pod "3f7eb4d7-8164-4074-be07-de089f528ca7" (UID: "3f7eb4d7-8164-4074-be07-de089f528ca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:49.124142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.124013 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce00f683-b3db-4b8a-8734-61b877751893-plugins-config-volume" pod "ce00f683-b3db-4b8a-8734-61b877751893" (UID: "ce00f683-b3db-4b8a-8734-61b877751893"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:49.124653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.124628 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7eb4d7-8164-4074-be07-de089f528ca7-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:49.124789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.124658 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce00f683-b3db-4b8a-8734-61b877751893-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:49.125689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.125666 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7eb4d7-8164-4074-be07-de089f528ca7-kube-api-access-t76tf" pod "3f7eb4d7-8164-4074-be07-de089f528ca7" (UID: "3f7eb4d7-8164-4074-be07-de089f528ca7"). InnerVolumeSpecName "kube-api-access-t76tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:49.125779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.125727 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce00f683-b3db-4b8a-8734-61b877751893-kube-api-access-nrknl" pod "ce00f683-b3db-4b8a-8734-61b877751893" (UID: "ce00f683-b3db-4b8a-8734-61b877751893"). InnerVolumeSpecName "kube-api-access-nrknl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:49.226101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.226006 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t76tf\" (UniqueName: \"kubernetes.io/projected/3f7eb4d7-8164-4074-be07-de089f528ca7-kube-api-access-t76tf\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:49.226101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.226042 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrknl\" (UniqueName: \"kubernetes.io/projected/ce00f683-b3db-4b8a-8734-61b877751893-kube-api-access-nrknl\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:49.351219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.351185 5515 generic.go:356] "Generic (PLEG): container finished" podID="3f7eb4d7-8164-4074-be07-de089f528ca7" containerID="222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22" exitCode=0 Oct 30 13:10:49.351439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.351234 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" Oct 30 13:10:49.351439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.351282 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" event={"ID":"3f7eb4d7-8164-4074-be07-de089f528ca7","Type":"ContainerDied","Data":"222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22"} Oct 30 13:10:49.351439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.351335 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt" event={"ID":"3f7eb4d7-8164-4074-be07-de089f528ca7","Type":"ContainerDied","Data":"4b383c56175182cb11c1cdf1ce57f92e00d59f5b71646d9c0033c5a32c6256d3"} Oct 30 13:10:49.351439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.351363 5515 scope.go:122] "RemoveContainer" containerID="222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22" Oct 30 13:10:49.352796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.352770 5515 generic.go:356] "Generic (PLEG): container finished" podID="ce00f683-b3db-4b8a-8734-61b877751893" containerID="a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92" exitCode=0 Oct 30 13:10:49.352913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.352804 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" event={"ID":"ce00f683-b3db-4b8a-8734-61b877751893","Type":"ContainerDied","Data":"a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92"} Oct 30 13:10:49.352913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.352871 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" event={"ID":"ce00f683-b3db-4b8a-8734-61b877751893","Type":"ContainerDied","Data":"dc63e8e8b51a6c399dec3de77d019446482e6e0713b4247387be9267e431deab"} Oct 30 13:10:49.352913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.352819 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb" Oct 30 13:10:49.362738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.362712 5515 scope.go:122] "RemoveContainer" containerID="222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22" Oct 30 13:10:49.363076 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:10:49.363055 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22\": container with ID starting with 222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22 not found: ID does not exist" containerID="222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22" Oct 30 13:10:49.363140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.363085 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22"} err="failed to get container status \"222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22\": rpc error: code = NotFound desc = could not find container \"222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22\": container with ID starting with 222027660b0653c7a808f9879622faf0e76511c4a4fe2f3ab1a1d03db2eefc22 not found: ID does not exist" Oct 30 13:10:49.363140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.363104 5515 scope.go:122] "RemoveContainer" containerID="a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92" Oct 30 13:10:49.372920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.372894 5515 scope.go:122] "RemoveContainer" containerID="a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92" Oct 30 13:10:49.373222 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:10:49.373204 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92\": container with ID starting with a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92 not found: ID does not exist" containerID="a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92" Oct 30 13:10:49.373295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.373231 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92"} err="failed to get container status \"a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92\": rpc error: code = NotFound desc = could not find container \"a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92\": container with ID starting with a43b1b0cb22527c64c25c81b2424c491b14daf0d458a5564836a01803f4fee92 not found: ID does not exist" Oct 30 13:10:49.407424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.407388 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb"] Oct 30 13:10:49.410121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.410091 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5ba2ef-68d98d98cf-2c9fb"] Oct 30 13:10:49.447020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.446986 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt"] Oct 30 13:10:49.449752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:49.449722 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5ba2ef-76df498cd6-lg7zt"] Oct 30 13:10:50.003694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:50.003615 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7eb4d7-8164-4074-be07-de089f528ca7" path="/var/lib/kubelet/pods/3f7eb4d7-8164-4074-be07-de089f528ca7/volumes" Oct 30 13:10:50.004073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:50.004057 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce00f683-b3db-4b8a-8734-61b877751893" path="/var/lib/kubelet/pods/ce00f683-b3db-4b8a-8734-61b877751893/volumes" Oct 30 13:10:50.417291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:50.413699 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:52.116759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:52.116708 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:53.599293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:53.599109 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:54.130498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:54.130461 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:54.838180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:54.838147 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd"] Oct 30 13:10:54.838945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:54.838907 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" podUID="2e7f38b2-319f-425e-aeb6-f465c3ec22ad" containerName="maas-api" containerID="cri-o://f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8" gracePeriod=30 Oct 30 13:10:54.996181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:54.996147 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:55.331524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.331485 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:10:55.364290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.358551 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-maas-api-tls\") pod \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " Oct 30 13:10:55.364290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.358628 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-metrics-certs\") pod \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " Oct 30 13:10:55.364290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.358766 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-kube-api-access-ql6gw\" (UniqueName: \"kubernetes.io/projected/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-kube-api-access-ql6gw\") pod \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " Oct 30 13:10:55.364290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.359047 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-openshift-service-ca\") pod \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\" (UID: \"2e7f38b2-319f-425e-aeb6-f465c3ec22ad\") " Oct 30 13:10:55.364290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.359945 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-openshift-service-ca" pod "2e7f38b2-319f-425e-aeb6-f465c3ec22ad" (UID: "2e7f38b2-319f-425e-aeb6-f465c3ec22ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:55.381253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.380433 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-maas-api-tls" pod "2e7f38b2-319f-425e-aeb6-f465c3ec22ad" (UID: "2e7f38b2-319f-425e-aeb6-f465c3ec22ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:55.441913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.441868 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-metrics-certs" pod "2e7f38b2-319f-425e-aeb6-f465c3ec22ad" (UID: "2e7f38b2-319f-425e-aeb6-f465c3ec22ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:10:55.442590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.442556 5515 generic.go:356] "Generic (PLEG): container finished" podID="2e7f38b2-319f-425e-aeb6-f465c3ec22ad" containerID="f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8" exitCode=0 Oct 30 13:10:55.442723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.442704 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" Oct 30 13:10:55.442786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.442723 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" event={"ID":"2e7f38b2-319f-425e-aeb6-f465c3ec22ad","Type":"ContainerDied","Data":"f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8"} Oct 30 13:10:55.442841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.442787 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd" event={"ID":"2e7f38b2-319f-425e-aeb6-f465c3ec22ad","Type":"ContainerDied","Data":"1e9a526cdc65065924de131b38b9ccc00c6bd460bc1ca8a0648da4a17973383a"} Oct 30 13:10:55.442841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.442818 5515 scope.go:122] "RemoveContainer" containerID="f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8" Oct 30 13:10:55.450529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.450489 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-kube-api-access-ql6gw" pod "2e7f38b2-319f-425e-aeb6-f465c3ec22ad" (UID: "2e7f38b2-319f-425e-aeb6-f465c3ec22ad"). InnerVolumeSpecName "kube-api-access-ql6gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:55.469497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.468487 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:55.469497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.468524 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:55.469497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.468538 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql6gw\" (UniqueName: \"kubernetes.io/projected/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-kube-api-access-ql6gw\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:55.469497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.468553 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7f38b2-319f-425e-aeb6-f465c3ec22ad-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:10:55.495958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.495859 5515 scope.go:122] "RemoveContainer" containerID="f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8" Oct 30 13:10:55.496427 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:10:55.496399 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8\": container with ID starting with f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8 not found: ID does not exist" containerID="f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8" Oct 30 13:10:55.496534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.496435 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8"} err="failed to get container status \"f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8\": rpc error: code = NotFound desc = could not find container \"f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8\": container with ID starting with f63917fef2cb690b13f8fbcac147c9e374e26414759ef652c31fa70be97915a8 not found: ID does not exist" Oct 30 13:10:55.533046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.533014 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:55.745356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.745256 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:55.792529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.792498 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd"] Oct 30 13:10:55.810914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.810880 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ba2ef-6648dfd7f4-lzqsd"] Oct 30 13:10:55.991959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:55.991920 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:56.004076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:56.004005 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7f38b2-319f-425e-aeb6-f465c3ec22ad" path="/var/lib/kubelet/pods/2e7f38b2-319f-425e-aeb6-f465c3ec22ad/volumes" Oct 30 13:10:56.204052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:56.204012 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:57.720139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:57.720105 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:58.987963 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:10:58.987927 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:10:59.545684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:10:59.545611 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:01.343294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:01.331769 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:02.396416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:02.396370 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:03.025044 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:03.025005 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:03.387501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:03.387450 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:03.612350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:03.609073 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:03.937217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:03.936831 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:03.968280 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:03.966886 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c"] Oct 30 13:11:03.968280 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:03.967206 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" containerID="cri-o://9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820" gracePeriod=30 Oct 30 13:11:04.153519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:04.153437 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:05.412296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:05.411879 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:05.425556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:05.425524 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:11:07.488436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:07.488396 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:11:07.489101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:07.489056 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:11:08.686586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.686536 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:08.714876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.714838 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:08.755007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.754962 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:08.848049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.847994 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:08.898372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.898330 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p"] Oct 30 13:11:08.899621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.899587 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" Oct 30 13:11:08.899808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.899792 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" Oct 30 13:11:08.899920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.899904 5515 cpu_manager.go:414] "containerMap: removing container" podUID="2e7f38b2-319f-425e-aeb6-f465c3ec22ad" containerName="maas-api" Oct 30 13:11:08.900008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.899996 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7f38b2-319f-425e-aeb6-f465c3ec22ad" containerName="maas-api" Oct 30 13:11:08.900127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900113 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ce00f683-b3db-4b8a-8734-61b877751893" containerName="payload-pre-processing" Oct 30 13:11:08.900230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900217 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce00f683-b3db-4b8a-8734-61b877751893" containerName="payload-pre-processing" Oct 30 13:11:08.900398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900385 5515 cpu_manager.go:414] "containerMap: removing container" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" Oct 30 13:11:08.900498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900486 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" Oct 30 13:11:08.900587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900576 5515 cpu_manager.go:414] "containerMap: removing container" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" Oct 30 13:11:08.900664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900652 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" Oct 30 13:11:08.900761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900751 5515 cpu_manager.go:414] "containerMap: removing container" podUID="dd36fae0-af4b-46c6-ac38-99740e6bd1c7" containerName="maas-api" Oct 30 13:11:08.900841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900831 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd36fae0-af4b-46c6-ac38-99740e6bd1c7" containerName="maas-api" Oct 30 13:11:08.900925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900915 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3f7eb4d7-8164-4074-be07-de089f528ca7" containerName="payload-processing" Oct 30 13:11:08.900999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.900989 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7eb4d7-8164-4074-be07-de089f528ca7" containerName="payload-processing" Oct 30 13:11:08.901088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.901077 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0bd6c2b6-791c-4133-9033-2ced2194d2a2" containerName="payload-processing" Oct 30 13:11:08.901170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.901158 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd6c2b6-791c-4133-9033-2ced2194d2a2" containerName="payload-processing" Oct 30 13:11:08.901468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.901447 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" Oct 30 13:11:08.901619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.901605 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" Oct 30 13:11:08.901726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.901714 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fddc62-e822-4b6f-be6f-870ac7691dac" containerName="istio-proxy" Oct 30 13:11:08.901827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.901816 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd36fae0-af4b-46c6-ac38-99740e6bd1c7" containerName="maas-api" Oct 30 13:11:08.901913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.901902 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7eb4d7-8164-4074-be07-de089f528ca7" containerName="payload-processing" Oct 30 13:11:08.901999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.901989 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd6c2b6-791c-4133-9033-2ced2194d2a2" containerName="payload-processing" Oct 30 13:11:08.902102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.902090 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7f38b2-319f-425e-aeb6-f465c3ec22ad" containerName="maas-api" Oct 30 13:11:08.902201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.902189 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce00f683-b3db-4b8a-8734-61b877751893" containerName="payload-pre-processing" Oct 30 13:11:08.918574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.918544 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:08.926652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.923654 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r"] Oct 30 13:11:08.926652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.924603 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" Oct 30 13:11:08.926652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.924623 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" Oct 30 13:11:08.926652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.924764 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9fce36-454e-4192-92ba-04acf3135f01" containerName="istio-proxy" Oct 30 13:11:08.949379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.932514 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p"] Oct 30 13:11:08.949379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.932641 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.019963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:08.985070 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:09.043794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.023005 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.043794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.023076 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c93a624-1411-42ef-a6b6-5a3dfba23243-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.043794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.023133 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.043794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.023189 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4785\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-kube-api-access-j4785\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.043794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.043475 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.043794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.043530 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.043794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.043593 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.043794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.043645 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.043794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.043685 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.045263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.044522 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.045263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.044596 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pcr2\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-kube-api-access-2pcr2\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.045263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.044628 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.045263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.044673 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.045263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.044742 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.045263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.044805 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.045263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.044845 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.045263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.044916 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.045263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.045120 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.150998 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4785\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-kube-api-access-j4785\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151068 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151109 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151147 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151177 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151210 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151294 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151326 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcr2\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-kube-api-access-2pcr2\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151355 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151381 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151425 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151467 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151502 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151548 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151676 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151725 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151778 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c93a624-1411-42ef-a6b6-5a3dfba23243-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.151818 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.152794 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.157802 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.179340 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.179616 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.180471 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.180964 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.207411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.181201 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.239330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.230889 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.239330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.231123 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.286879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.266701 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c93a624-1411-42ef-a6b6-5a3dfba23243-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.286879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.271012 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r"] Oct 30 13:11:09.326081 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.326020 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.409092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.350571 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.409092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.351247 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.409092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.354003 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.409092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.370461 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4785\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-kube-api-access-j4785\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.409092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.370996 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.409092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.371191 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcr2\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-kube-api-access-2pcr2\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.409092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.407957 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.426300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.420978 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.558805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.558545 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:09.601468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.601440 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:11:09.683527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.683485 5515 generic.go:356] "Generic (PLEG): container finished" podID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerID="9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820" exitCode=0 Oct 30 13:11:09.683733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.683544 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" Oct 30 13:11:09.683733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.683653 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" event={"ID":"c08af005-2812-4d6c-97d0-c7aea99250bd","Type":"ContainerDied","Data":"9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820"} Oct 30 13:11:09.683733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.683693 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c" event={"ID":"c08af005-2812-4d6c-97d0-c7aea99250bd","Type":"ContainerDied","Data":"727020f48713a81223606ae906bc72d1719e2682e55a3517bd7e46f9dcbb82bc"} Oct 30 13:11:09.683733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.683718 5515 scope.go:122] "RemoveContainer" containerID="9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820" Oct 30 13:11:09.690073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.690047 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-podinfo\") pod \"c08af005-2812-4d6c-97d0-c7aea99250bd\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " Oct 30 13:11:09.690492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.690125 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-envoy\") pod \"c08af005-2812-4d6c-97d0-c7aea99250bd\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " Oct 30 13:11:09.690492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.690160 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-socket\") pod \"c08af005-2812-4d6c-97d0-c7aea99250bd\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " Oct 30 13:11:09.690492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.690216 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-credential-socket\") pod \"c08af005-2812-4d6c-97d0-c7aea99250bd\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " Oct 30 13:11:09.690492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.690252 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-kube-api-access-n5lrk\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-kube-api-access-n5lrk\") pod \"c08af005-2812-4d6c-97d0-c7aea99250bd\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " Oct 30 13:11:09.690492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.690319 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08af005-2812-4d6c-97d0-c7aea99250bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08af005-2812-4d6c-97d0-c7aea99250bd-istiod-ca-cert\") pod \"c08af005-2812-4d6c-97d0-c7aea99250bd\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " Oct 30 13:11:09.690492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.690351 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-data\") pod \"c08af005-2812-4d6c-97d0-c7aea99250bd\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " Oct 30 13:11:09.690492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.690401 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-token\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-token\") pod \"c08af005-2812-4d6c-97d0-c7aea99250bd\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " Oct 30 13:11:09.690492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.690445 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-certs\") pod \"c08af005-2812-4d6c-97d0-c7aea99250bd\" (UID: \"c08af005-2812-4d6c-97d0-c7aea99250bd\") " Oct 30 13:11:09.691381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.691355 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-certs" pod "c08af005-2812-4d6c-97d0-c7aea99250bd" (UID: "c08af005-2812-4d6c-97d0-c7aea99250bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:09.691945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.691681 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08af005-2812-4d6c-97d0-c7aea99250bd-istiod-ca-cert" pod "c08af005-2812-4d6c-97d0-c7aea99250bd" (UID: "c08af005-2812-4d6c-97d0-c7aea99250bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:09.691945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.691719 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-socket" pod "c08af005-2812-4d6c-97d0-c7aea99250bd" (UID: "c08af005-2812-4d6c-97d0-c7aea99250bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:09.691945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.691726 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-data" pod "c08af005-2812-4d6c-97d0-c7aea99250bd" (UID: "c08af005-2812-4d6c-97d0-c7aea99250bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:09.691945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.691919 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-credential-socket" pod "c08af005-2812-4d6c-97d0-c7aea99250bd" (UID: "c08af005-2812-4d6c-97d0-c7aea99250bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:09.695022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.693747 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-podinfo" pod "c08af005-2812-4d6c-97d0-c7aea99250bd" (UID: "c08af005-2812-4d6c-97d0-c7aea99250bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:09.695022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.694102 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-kube-api-access-n5lrk" pod "c08af005-2812-4d6c-97d0-c7aea99250bd" (UID: "c08af005-2812-4d6c-97d0-c7aea99250bd"). InnerVolumeSpecName "kube-api-access-n5lrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:09.696837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.696815 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-token" pod "c08af005-2812-4d6c-97d0-c7aea99250bd" (UID: "c08af005-2812-4d6c-97d0-c7aea99250bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:09.698175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.698154 5515 scope.go:122] "RemoveContainer" containerID="0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500" Oct 30 13:11:09.698379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.698343 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-envoy" pod "c08af005-2812-4d6c-97d0-c7aea99250bd" (UID: "c08af005-2812-4d6c-97d0-c7aea99250bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:09.700252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.700234 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:09.714674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.713720 5515 scope.go:122] "RemoveContainer" containerID="9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820" Oct 30 13:11:09.714674 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:11:09.714050 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820\": container with ID starting with 9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820 not found: ID does not exist" containerID="9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820" Oct 30 13:11:09.714674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.714083 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820"} err="failed to get container status \"9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820\": rpc error: code = NotFound desc = could not find container \"9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820\": container with ID starting with 9631f69aa4839a6469c55076f09ada9500f33eda4380e313a80ed8d00473f820 not found: ID does not exist" Oct 30 13:11:09.714674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.714120 5515 scope.go:122] "RemoveContainer" containerID="0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500" Oct 30 13:11:09.714674 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:11:09.714477 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500\": container with ID starting with 0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500 not found: ID does not exist" containerID="0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500" Oct 30 13:11:09.714674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.714526 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500"} err="failed to get container status \"0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500\": rpc error: code = NotFound desc = could not find container \"0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500\": container with ID starting with 0711cbe3287d5a328bef1fccdcba1ff6fed99ece6b171686e132e789e6909500 not found: ID does not exist" Oct 30 13:11:09.792460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.792431 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.792460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.792458 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lrk\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-kube-api-access-n5lrk\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.792460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.792469 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08af005-2812-4d6c-97d0-c7aea99250bd-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.792712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.792478 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.792712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.792486 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.792712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.792494 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.792712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.792501 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.792712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.792510 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.792712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.792523 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08af005-2812-4d6c-97d0-c7aea99250bd-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:09.893850 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:11:09.893805 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c93a624_1411_42ef_a6b6_5a3dfba23243.slice/crio-fcc31ef5ffc01ea91dde36adeaad2ed39a6cd3ef60c5e083fdff4c4e2e744c42 WatchSource:0}: Error finding container fcc31ef5ffc01ea91dde36adeaad2ed39a6cd3ef60c5e083fdff4c4e2e744c42: Status 404 returned error can't find the container with id fcc31ef5ffc01ea91dde36adeaad2ed39a6cd3ef60c5e083fdff4c4e2e744c42 Oct 30 13:11:09.893850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.893828 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p"] Oct 30 13:11:09.896112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.896079 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:09.896209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.896144 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:09.896209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:09.896174 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:10.034042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.033961 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r"] Oct 30 13:11:10.043065 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.043006 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:10.043261 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.043242 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:10.043383 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.043370 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:10.049333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.049302 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c"] Oct 30 13:11:10.051868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.051844 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5ba2ef-openshift-default-555b66d977-4f27c"] Oct 30 13:11:10.692364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.692304 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" event={"ID":"4c93a624-1411-42ef-a6b6-5a3dfba23243","Type":"ContainerStarted","Data":"ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d"} Oct 30 13:11:10.692851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.692375 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" event={"ID":"4c93a624-1411-42ef-a6b6-5a3dfba23243","Type":"ContainerStarted","Data":"fcc31ef5ffc01ea91dde36adeaad2ed39a6cd3ef60c5e083fdff4c4e2e744c42"} Oct 30 13:11:10.695905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.695856 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" event={"ID":"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8","Type":"ContainerStarted","Data":"90d1803e29a61fbe48b6878d4849ee9827abf50df2684aaab1894f099b002105"} Oct 30 13:11:10.696072 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.695929 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" event={"ID":"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8","Type":"ContainerStarted","Data":"90c23b4c5ba98826ef80696e4fff596f79855c42a3d936a74befa71b071bca1f"} Oct 30 13:11:10.701195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.701172 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:10.827514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.827457 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" podStartSLOduration=2.827443642 podStartE2EDuration="2.827443642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:10.777230206 +0000 UTC m=+2288.904031108" watchObservedRunningTime="2026-10-30 13:11:10.827443642 +0000 UTC m=+2288.954244540" Oct 30 13:11:10.829559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.829516 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" podStartSLOduration=2.8295037 podStartE2EDuration="2.8295037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:10.826465223 +0000 UTC m=+2288.953266151" watchObservedRunningTime="2026-10-30 13:11:10.8295037 +0000 UTC m=+2288.956304597" Oct 30 13:11:10.989804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:10.989695 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:11.563385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:11.563343 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:11.565195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:11.565164 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:11.565377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:11.565239 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:11:11.795294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:11.792434 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:11:11.795294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:11.792494 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:11:11.819363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:11.817978 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:12.058285 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:12.058231 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" path="/var/lib/kubelet/pods/c08af005-2812-4d6c-97d0-c7aea99250bd/volumes" Oct 30 13:11:12.562824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:12.562778 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:12.562999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:12.562839 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:11:12.708163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:12.708125 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:11:12.708459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:12.708428 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:11:13.316327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:13.316191 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:13.563685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:13.563641 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:13.563874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:13.563715 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:11:13.747295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:13.745036 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:11:13.747295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:13.745104 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:11:14.564441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:14.564401 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:14.564824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:14.564577 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:11:14.708445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:14.708401 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:11:14.708752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:14.708717 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:11:15.000294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:14.997481 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:15.216326 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:15.216017 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:15.453293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:15.451768 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:15.563567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:15.563527 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:15.563756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:15.563587 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:11:15.753358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:15.751566 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:11:15.753358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:15.751639 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:11:16.562942 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:16.562905 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 13:11:16.563098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:16.562961 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 13:11:16.659331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:16.659282 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:16.727293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:16.718580 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 13:11:16.727293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:16.718637 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 13:11:17.640949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:17.640895 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:17.863090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:17.863041 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:18.015645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.015559 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:18.015950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.015926 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:18.016958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.016936 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:11:18.102738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.101421 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:18.102738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.102657 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:18.105627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.105596 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:11:18.249312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.249248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:18.689315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.689257 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf"] Oct 30 13:11:18.690046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.690016 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:18.928601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.928570 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:18.978444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:18.978344 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" podUID="3923d62a-9587-4c06-a628-2a7f012e0540" containerName="payload-processing" containerID="cri-o://dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4" gracePeriod=30 Oct 30 13:11:19.335571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.335533 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:19.373201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.373169 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:11:19.515555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.515518 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3923d62a-9587-4c06-a628-2a7f012e0540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923d62a-9587-4c06-a628-2a7f012e0540-plugins-config-volume\") pod \"3923d62a-9587-4c06-a628-2a7f012e0540\" (UID: \"3923d62a-9587-4c06-a628-2a7f012e0540\") " Oct 30 13:11:19.515748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.515578 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3923d62a-9587-4c06-a628-2a7f012e0540-kube-api-access-8h7ll\" (UniqueName: \"kubernetes.io/projected/3923d62a-9587-4c06-a628-2a7f012e0540-kube-api-access-8h7ll\") pod \"3923d62a-9587-4c06-a628-2a7f012e0540\" (UID: \"3923d62a-9587-4c06-a628-2a7f012e0540\") " Oct 30 13:11:19.515926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.515904 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3923d62a-9587-4c06-a628-2a7f012e0540-plugins-config-volume" pod "3923d62a-9587-4c06-a628-2a7f012e0540" (UID: "3923d62a-9587-4c06-a628-2a7f012e0540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:19.516121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.516105 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923d62a-9587-4c06-a628-2a7f012e0540-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:19.517775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.517753 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3923d62a-9587-4c06-a628-2a7f012e0540-kube-api-access-8h7ll" pod "3923d62a-9587-4c06-a628-2a7f012e0540" (UID: "3923d62a-9587-4c06-a628-2a7f012e0540"). InnerVolumeSpecName "kube-api-access-8h7ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:19.561005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.560840 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:19.617354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.617317 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h7ll\" (UniqueName: \"kubernetes.io/projected/3923d62a-9587-4c06-a628-2a7f012e0540-kube-api-access-8h7ll\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:19.853174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.853089 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:19.985632 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.985584 5515 generic.go:356] "Generic (PLEG): container finished" podID="3923d62a-9587-4c06-a628-2a7f012e0540" containerID="dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4" exitCode=0 Oct 30 13:11:19.985632 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.985634 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" Oct 30 13:11:19.985906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.985656 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" event={"ID":"3923d62a-9587-4c06-a628-2a7f012e0540","Type":"ContainerDied","Data":"dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4"} Oct 30 13:11:19.985906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.985726 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf" event={"ID":"3923d62a-9587-4c06-a628-2a7f012e0540","Type":"ContainerDied","Data":"45b7fd71499cfb4722b7c1a066f1db2017e238098e364057b9c91664550c0849"} Oct 30 13:11:19.985906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.985756 5515 scope.go:122] "RemoveContainer" containerID="dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4" Oct 30 13:11:19.996459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.996434 5515 scope.go:122] "RemoveContainer" containerID="dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4" Oct 30 13:11:19.996806 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:11:19.996782 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4\": container with ID starting with dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4 not found: ID does not exist" containerID="dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4" Oct 30 13:11:19.996866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:19.996816 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4"} err="failed to get container status \"dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4\": rpc error: code = NotFound desc = could not find container \"dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4\": container with ID starting with dad71cac54bf4efb08e6fc390e9f830804bc3a4ba4423e784b89de03942f9af4 not found: ID does not exist" Oct 30 13:11:20.045111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:20.045073 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf"] Oct 30 13:11:20.050528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:20.050497 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-86c5c947d8-dh4hf"] Oct 30 13:11:20.110902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:20.110813 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:21.385298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:21.381827 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:22.001763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:22.001733 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3923d62a-9587-4c06-a628-2a7f012e0540" path="/var/lib/kubelet/pods/3923d62a-9587-4c06-a628-2a7f012e0540/volumes" Oct 30 13:11:23.002290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:23.000555 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:24.534428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:24.534391 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:26.170291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.169380 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:26.603853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.601933 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:26.662283 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.662241 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks"] Oct 30 13:11:26.662978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.662957 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3923d62a-9587-4c06-a628-2a7f012e0540" containerName="payload-processing" Oct 30 13:11:26.662978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.662979 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3923d62a-9587-4c06-a628-2a7f012e0540" containerName="payload-processing" Oct 30 13:11:26.663135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.662994 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" Oct 30 13:11:26.663135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.662999 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" Oct 30 13:11:26.663135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.663009 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" Oct 30 13:11:26.663135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.663014 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" Oct 30 13:11:26.663135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.663120 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3923d62a-9587-4c06-a628-2a7f012e0540" containerName="payload-processing" Oct 30 13:11:26.663135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.663128 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" Oct 30 13:11:26.663135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.663136 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08af005-2812-4d6c-97d0-c7aea99250bd" containerName="istio-proxy" Oct 30 13:11:26.666586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.666566 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.693216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.693179 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks"] Oct 30 13:11:26.722478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.722437 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.722681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.722590 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.722681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.722630 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc530b2d-6082-47d9-b62a-09593698d456-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.722792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.722721 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc530b2d-6082-47d9-b62a-09593698d456-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.722864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.722844 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.723006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.722985 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.723052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.723033 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.723133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.723113 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.723175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.723148 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv49h\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-kube-api-access-sv49h\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.824983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.824942 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.824983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.824984 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.825221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825022 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.825221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825046 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv49h\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-kube-api-access-sv49h\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.825221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825145 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.825454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825237 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.825454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825289 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc530b2d-6082-47d9-b62a-09593698d456-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.825454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825319 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc530b2d-6082-47d9-b62a-09593698d456-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.825454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825388 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.825971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825946 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.826155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825676 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.826258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.826148 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.826412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.825753 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.826543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.826518 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc530b2d-6082-47d9-b62a-09593698d456-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.830252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.830152 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc530b2d-6082-47d9-b62a-09593698d456-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.838158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.838114 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:26.851975 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.851938 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.861376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.852617 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv49h\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-kube-api-access-sv49h\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.863527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.862499 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:26.999665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:26.983005 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:27.084731 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:27.084697 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:27.432586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:27.432557 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks"] Oct 30 13:11:27.432946 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:11:27.432561 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc530b2d_6082_47d9_b62a_09593698d456.slice/crio-9a51ea862c5b4c0d204d14b971822f9ce17bc554aee6c2d66b87b10a1363511d WatchSource:0}: Error finding container 9a51ea862c5b4c0d204d14b971822f9ce17bc554aee6c2d66b87b10a1363511d: Status 404 returned error can't find the container with id 9a51ea862c5b4c0d204d14b971822f9ce17bc554aee6c2d66b87b10a1363511d Oct 30 13:11:27.435239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:27.435206 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:27.435378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:27.435291 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:27.435378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:27.435326 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:27.904645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:27.902238 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:28.056949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.056908 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" event={"ID":"cc530b2d-6082-47d9-b62a-09593698d456","Type":"ContainerStarted","Data":"96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763"} Oct 30 13:11:28.057141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.056973 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" event={"ID":"cc530b2d-6082-47d9-b62a-09593698d456","Type":"ContainerStarted","Data":"9a51ea862c5b4c0d204d14b971822f9ce17bc554aee6c2d66b87b10a1363511d"} Oct 30 13:11:28.107851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.107794 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podStartSLOduration=2.107779153 podStartE2EDuration="2.107779153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:28.106457197 +0000 UTC m=+2306.233258094" watchObservedRunningTime="2026-10-30 13:11:28.107779153 +0000 UTC m=+2306.234580131" Oct 30 13:11:28.144303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.142821 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:28.365659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.365563 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx"] Oct 30 13:11:28.365835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.365807 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" containerID="cri-o://5fab7c1e6fc1d129d85475fcc61658021a8dc419487052f467ab8d55df4c2e18" gracePeriod=30 Oct 30 13:11:28.487290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.484072 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:28.985185 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.984804 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:28.987043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.987012 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:11:28.987177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:28.987071 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:11:28.988683 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:11:28.988653 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:11:29.951937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:29.950657 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:29.984772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:29.984732 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:11:29.984960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:29.984802 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:11:30.156566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.156521 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:30.435089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.435051 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl"] Oct 30 13:11:30.439938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.439913 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.471157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.471120 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl"] Oct 30 13:11:30.474257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.474227 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-2fdf91-openshift-default-dockercfg-55xpk\"" Oct 30 13:11:30.651821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.651774 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff0adac-2f41-4528-ab90-ff59a93db902-istiod-ca-cert\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.652123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.652097 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-credential-socket\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.652388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.652367 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-envoy\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.652576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.652561 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-certs\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.652698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.652681 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vtj\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-kube-api-access-55vtj\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.652869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.652852 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-token\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.653006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.652991 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-data\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.653112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.653097 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-podinfo\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.653224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.653208 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-socket\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.690415 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.690232 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.754428 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff0adac-2f41-4528-ab90-ff59a93db902-istiod-ca-cert\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.754495 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-credential-socket\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.754599 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-envoy\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.754675 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-certs\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.754714 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vtj\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-kube-api-access-55vtj\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.754792 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-token\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.754861 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-data\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.754888 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-podinfo\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.754926 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-socket\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.755457 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-socket\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.756109 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff0adac-2f41-4528-ab90-ff59a93db902-istiod-ca-cert\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.756372 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-credential-socket\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.757099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.756740 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-data\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.763059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.763037 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-certs\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.835183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.835111 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-podinfo\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.851292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.849805 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-token\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.851292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.850054 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-envoy\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.852636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.852611 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vtj\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-kube-api-access-55vtj\") pod \"e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:30.915736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.915701 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:30.984668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.984575 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:11:30.984668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:30.984638 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:11:31.059992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:31.059953 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:31.421788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:31.421749 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl"] Oct 30 13:11:31.421992 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:11:31.421910 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff0adac_2f41_4528_ab90_ff59a93db902.slice/crio-58d73640014f590727f65ca84b47634018c01511c147e85215cef9c3383e95ff WatchSource:0}: Error finding container 58d73640014f590727f65ca84b47634018c01511c147e85215cef9c3383e95ff: Status 404 returned error can't find the container with id 58d73640014f590727f65ca84b47634018c01511c147e85215cef9c3383e95ff Oct 30 13:11:31.424608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:31.424568 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:31.424712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:31.424642 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:31.424712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:31.424678 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:31.984607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:31.984562 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:11:31.984832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:31.984619 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:11:32.067049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:32.066943 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:32.132071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:32.130672 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" event={"ID":"1ff0adac-2f41-4528-ab90-ff59a93db902","Type":"ContainerStarted","Data":"b8097d89590850f890f2e2a320c0b126a706f1daf2b5a5428896aab65133e1aa"} Oct 30 13:11:32.132071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:32.130745 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" event={"ID":"1ff0adac-2f41-4528-ab90-ff59a93db902","Type":"ContainerStarted","Data":"58d73640014f590727f65ca84b47634018c01511c147e85215cef9c3383e95ff"} Oct 30 13:11:32.181326 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:32.181233 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" podStartSLOduration=2.181217279 podStartE2EDuration="2.181217279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:32.179307524 +0000 UTC m=+2310.306108421" watchObservedRunningTime="2026-10-30 13:11:32.181217279 +0000 UTC m=+2310.308018178" Oct 30 13:11:32.953422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:32.953380 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:11:32.985155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:32.985118 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:11:32.985618 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:32.985188 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:11:33.060510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:33.060470 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:33.065681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:33.065658 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:33.136307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:33.136260 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:33.137120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:33.137102 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:11:33.987706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:33.987661 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:11:33.988825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:33.988791 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:11:34.313108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.312993 5515 generic.go:356] "Generic (PLEG): container finished" podID="5b15247e-2417-4458-aada-9598ad342b4e" containerID="5fab7c1e6fc1d129d85475fcc61658021a8dc419487052f467ab8d55df4c2e18" exitCode=0 Oct 30 13:11:34.314661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.314630 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" event={"ID":"5b15247e-2417-4458-aada-9598ad342b4e","Type":"ContainerDied","Data":"5fab7c1e6fc1d129d85475fcc61658021a8dc419487052f467ab8d55df4c2e18"} Oct 30 13:11:34.315604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.315580 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" event={"ID":"5b15247e-2417-4458-aada-9598ad342b4e","Type":"ContainerDied","Data":"94fc4e2b4a880872fde59e82d3737ed9faa498fdfa92bbda27883b208285cc9b"} Oct 30 13:11:34.315847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.315829 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94fc4e2b4a880872fde59e82d3737ed9faa498fdfa92bbda27883b208285cc9b" Oct 30 13:11:34.315963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.315949 5515 scope.go:122] "RemoveContainer" containerID="2c64d78b327475d74da78e143a2d6078b03e5a1ecfce682b528e6c4c413bd4e4" Oct 30 13:11:34.334724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.334674 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:11:34.478818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.478768 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b15247e-2417-4458-aada-9598ad342b4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b15247e-2417-4458-aada-9598ad342b4e-istio-podinfo\") pod \"5b15247e-2417-4458-aada-9598ad342b4e\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " Oct 30 13:11:34.480323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.480295 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-envoy\") pod \"5b15247e-2417-4458-aada-9598ad342b4e\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " Oct 30 13:11:34.486593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.486514 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b15247e-2417-4458-aada-9598ad342b4e-istiod-ca-cert" pod "5b15247e-2417-4458-aada-9598ad342b4e" (UID: "5b15247e-2417-4458-aada-9598ad342b4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:34.486839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.486816 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b15247e-2417-4458-aada-9598ad342b4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b15247e-2417-4458-aada-9598ad342b4e-istiod-ca-cert\") pod \"5b15247e-2417-4458-aada-9598ad342b4e\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " Oct 30 13:11:34.527463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.527424 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-istio-token\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-istio-token\") pod \"5b15247e-2417-4458-aada-9598ad342b4e\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " Oct 30 13:11:34.559579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.559534 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-credential-socket\") pod \"5b15247e-2417-4458-aada-9598ad342b4e\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " Oct 30 13:11:34.637343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.613110 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-certs\") pod \"5b15247e-2417-4458-aada-9598ad342b4e\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " Oct 30 13:11:34.637657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.576394 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-credential-socket" pod "5b15247e-2417-4458-aada-9598ad342b4e" (UID: "5b15247e-2417-4458-aada-9598ad342b4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:34.637657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.622596 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-certs" pod "5b15247e-2417-4458-aada-9598ad342b4e" (UID: "5b15247e-2417-4458-aada-9598ad342b4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:34.637657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.637126 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-envoy" pod "5b15247e-2417-4458-aada-9598ad342b4e" (UID: "5b15247e-2417-4458-aada-9598ad342b4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:34.649632 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.649592 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-socket\") pod \"5b15247e-2417-4458-aada-9598ad342b4e\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " Oct 30 13:11:34.708897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.699779 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-kube-api-access-n4rqb\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-kube-api-access-n4rqb\") pod \"5b15247e-2417-4458-aada-9598ad342b4e\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " Oct 30 13:11:34.709232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.709199 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-data\") pod \"5b15247e-2417-4458-aada-9598ad342b4e\" (UID: \"5b15247e-2417-4458-aada-9598ad342b4e\") " Oct 30 13:11:34.716304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.710163 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:34.726143 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.726113 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b15247e-2417-4458-aada-9598ad342b4e-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:34.735638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.735609 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:34.735796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.735779 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:34.735897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.662520 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-istio-token" pod "5b15247e-2417-4458-aada-9598ad342b4e" (UID: "5b15247e-2417-4458-aada-9598ad342b4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:34.735987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.684396 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-socket" pod "5b15247e-2417-4458-aada-9598ad342b4e" (UID: "5b15247e-2417-4458-aada-9598ad342b4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:34.736094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.707439 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b15247e-2417-4458-aada-9598ad342b4e-istio-podinfo" pod "5b15247e-2417-4458-aada-9598ad342b4e" (UID: "5b15247e-2417-4458-aada-9598ad342b4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:11:34.736181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.725384 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-data" pod "5b15247e-2417-4458-aada-9598ad342b4e" (UID: "5b15247e-2417-4458-aada-9598ad342b4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:11:34.780359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.780262 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-kube-api-access-n4rqb" pod "5b15247e-2417-4458-aada-9598ad342b4e" (UID: "5b15247e-2417-4458-aada-9598ad342b4e"). InnerVolumeSpecName "kube-api-access-n4rqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:34.837513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.837480 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:34.837756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.837739 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b15247e-2417-4458-aada-9598ad342b4e-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:34.837884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.837868 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:34.837988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.837974 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b15247e-2417-4458-aada-9598ad342b4e-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:34.838097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.838082 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4rqb\" (UniqueName: \"kubernetes.io/projected/5b15247e-2417-4458-aada-9598ad342b4e-kube-api-access-n4rqb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:34.991822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.991725 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:11:34.992461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:34.992420 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:11:35.241291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:35.236006 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:35.386291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:35.384750 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx" Oct 30 13:11:35.527296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:35.521160 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:35.640357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:35.624154 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx"] Oct 30 13:11:35.700595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:35.700535 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-76c107-openshift-default-69f98c49-8gnzx"] Oct 30 13:11:35.774909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:35.774876 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:35.807980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:35.807936 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:35.993017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:35.992908 5515 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 13:11:35.993680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:35.993629 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 13:11:36.061163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:36.061124 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b15247e-2417-4458-aada-9598ad342b4e" path="/var/lib/kubelet/pods/5b15247e-2417-4458-aada-9598ad342b4e/volumes" Oct 30 13:11:36.105178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:36.105064 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded" Oct 30 13:11:36.449452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:36.449392 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:36.706352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:36.702092 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:36.835085 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:36.820760 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:36.872429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:36.872396 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 13:11:37.069393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:37.052337 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:37.651243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:37.651194 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:37.691740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:37.691708 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 13:11:37.727138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:37.727107 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 13:11:37.735592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:37.735564 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 13:11:37.900318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:37.900243 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:37.901950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:37.901880 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:37.905490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:37.905459 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:11:38.331060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:38.330969 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:38.583948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:38.583859 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:39.155982 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:39.155925 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:41.004339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.004301 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:41.165559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.165524 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b"] Oct 30 13:11:41.166180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.166157 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" Oct 30 13:11:41.166180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.166176 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" Oct 30 13:11:41.166431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.166321 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" Oct 30 13:11:41.166431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.166333 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" Oct 30 13:11:41.170916 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.170898 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.193779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.193744 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b"] Oct 30 13:11:41.201770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.201732 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:11:41.201922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.201815 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\"" Oct 30 13:11:41.287885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.287778 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swcfl\" (UniqueName: \"kubernetes.io/projected/60244c12-b78e-4179-bc95-f3da973e6421-kube-api-access-swcfl\") pod \"maas-api-revoke-sub-eadbb5ef-mr55b\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.287885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.287856 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eadbb5ef-mr55b\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.288144 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.287964 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eadbb5ef-mr55b\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.389966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.389929 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swcfl\" (UniqueName: \"kubernetes.io/projected/60244c12-b78e-4179-bc95-f3da973e6421-kube-api-access-swcfl\") pod \"maas-api-revoke-sub-eadbb5ef-mr55b\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.390190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.390022 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eadbb5ef-mr55b\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.390190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.390073 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eadbb5ef-mr55b\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.390787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.390760 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eadbb5ef-mr55b\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.392690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.392660 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eadbb5ef-mr55b\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.414223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.414188 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swcfl\" (UniqueName: \"kubernetes.io/projected/60244c12-b78e-4179-bc95-f3da973e6421-kube-api-access-swcfl\") pod \"maas-api-revoke-sub-eadbb5ef-mr55b\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.485256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.485219 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:41.785875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.785840 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b"] Oct 30 13:11:41.786328 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:11:41.786305 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60244c12_b78e_4179_bc95_f3da973e6421.slice/crio-6ce4d401bf369096cadc8357b9e417b63b89933b4764ed340ea705c9be11be4d WatchSource:0}: Error finding container 6ce4d401bf369096cadc8357b9e417b63b89933b4764ed340ea705c9be11be4d: Status 404 returned error can't find the container with id 6ce4d401bf369096cadc8357b9e417b63b89933b4764ed340ea705c9be11be4d Oct 30 13:11:41.851944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:41.851907 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" event={"ID":"60244c12-b78e-4179-bc95-f3da973e6421","Type":"ContainerStarted","Data":"6ce4d401bf369096cadc8357b9e417b63b89933b4764ed340ea705c9be11be4d"} Oct 30 13:11:42.666895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:42.666566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:42.859315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:42.859249 5515 generic.go:356] "Generic (PLEG): container finished" podID="60244c12-b78e-4179-bc95-f3da973e6421" containerID="569b5b699c1bbb9cdad0871fac0cd22b70c7919065400c46b081c18a0082c525" exitCode=0 Oct 30 13:11:42.859315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:42.859299 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" event={"ID":"60244c12-b78e-4179-bc95-f3da973e6421","Type":"ContainerDied","Data":"569b5b699c1bbb9cdad0871fac0cd22b70c7919065400c46b081c18a0082c525"} Oct 30 13:11:44.031757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.031730 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:44.153324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.153262 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-xks-ca\") pod \"60244c12-b78e-4179-bc95-f3da973e6421\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " Oct 30 13:11:44.153586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.153567 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60244c12-b78e-4179-bc95-f3da973e6421-kube-api-access-swcfl\" (UniqueName: \"kubernetes.io/projected/60244c12-b78e-4179-bc95-f3da973e6421-kube-api-access-swcfl\") pod \"60244c12-b78e-4179-bc95-f3da973e6421\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " Oct 30 13:11:44.153660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.153645 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-openshift-ca\") pod \"60244c12-b78e-4179-bc95-f3da973e6421\" (UID: \"60244c12-b78e-4179-bc95-f3da973e6421\") " Oct 30 13:11:44.154088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.154056 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-openshift-ca" pod "60244c12-b78e-4179-bc95-f3da973e6421" (UID: "60244c12-b78e-4179-bc95-f3da973e6421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:11:44.154550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.154433 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-openshift-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:44.155710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.155687 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-xks-ca" pod "60244c12-b78e-4179-bc95-f3da973e6421" (UID: "60244c12-b78e-4179-bc95-f3da973e6421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:11:44.155818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.155711 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60244c12-b78e-4179-bc95-f3da973e6421-kube-api-access-swcfl" pod "60244c12-b78e-4179-bc95-f3da973e6421" (UID: "60244c12-b78e-4179-bc95-f3da973e6421"). InnerVolumeSpecName "kube-api-access-swcfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:11:44.256213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.256176 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swcfl\" (UniqueName: \"kubernetes.io/projected/60244c12-b78e-4179-bc95-f3da973e6421-kube-api-access-swcfl\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:44.256213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.256208 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60244c12-b78e-4179-bc95-f3da973e6421-maas-api-xks-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:11:44.511819 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.510459 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:44.874550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.874455 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" event={"ID":"60244c12-b78e-4179-bc95-f3da973e6421","Type":"ContainerDied","Data":"6ce4d401bf369096cadc8357b9e417b63b89933b4764ed340ea705c9be11be4d"} Oct 30 13:11:44.874550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.874509 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce4d401bf369096cadc8357b9e417b63b89933b4764ed340ea705c9be11be4d" Oct 30 13:11:44.874550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:44.874470 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b" Oct 30 13:11:45.171928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:45.171882 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:45.706415 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:45.706379 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:47.151471 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.151437 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:47.696482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.696444 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6"] Oct 30 13:11:47.697182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.697161 5515 cpu_manager.go:414] "containerMap: removing container" podUID="60244c12-b78e-4179-bc95-f3da973e6421" containerName="revoke-keys" Oct 30 13:11:47.697306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.697183 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60244c12-b78e-4179-bc95-f3da973e6421" containerName="revoke-keys" Oct 30 13:11:47.697306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.697204 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" Oct 30 13:11:47.697306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.697211 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b15247e-2417-4458-aada-9598ad342b4e" containerName="istio-proxy" Oct 30 13:11:47.697474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.697419 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="60244c12-b78e-4179-bc95-f3da973e6421" containerName="revoke-keys" Oct 30 13:11:47.701169 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.701143 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.719108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.719076 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-2fdf91-serving-cert\"" Oct 30 13:11:47.734632 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.734594 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6"] Oct 30 13:11:47.744338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.744299 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.744806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.744775 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c75c816-7912-4460-b898-ad9da761027b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.745105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.745081 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-metrics-certs\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.745213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.745136 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d5lz\" (UniqueName: \"kubernetes.io/projected/0c75c816-7912-4460-b898-ad9da761027b-kube-api-access-8d5lz\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.846571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.846530 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c75c816-7912-4460-b898-ad9da761027b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.846778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.846616 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-metrics-certs\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.846778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.846656 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5lz\" (UniqueName: \"kubernetes.io/projected/0c75c816-7912-4460-b898-ad9da761027b-kube-api-access-8d5lz\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.846778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.846725 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.846963 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:11:47.846924 5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-serving-cert: secret "maas-api-e2e-praxis-a-2fdf91-serving-cert" not found Oct 30 13:11:47.847025 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:11:47.846991 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls podName:0c75c816-7912-4460-b898-ad9da761027b nodeName:}" failed. No retries permitted until 2026-10-30 13:11:48.346973457 +0000 UTC m=+2326.473774332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls") pod "maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" (UID: "0c75c816-7912-4460-b898-ad9da761027b") : secret "maas-api-e2e-praxis-a-2fdf91-serving-cert" not found Oct 30 13:11:47.847457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.847424 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c75c816-7912-4460-b898-ad9da761027b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.849201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.849182 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-metrics-certs\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.900755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.893004 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5lz\" (UniqueName: \"kubernetes.io/projected/0c75c816-7912-4460-b898-ad9da761027b-kube-api-access-8d5lz\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:47.926130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:47.925829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:48.125382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:48.125287 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:48.359971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:48.359929 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:48.362390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:48.362365 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:48.617059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:48.616247 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:48.923766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:48.923734 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6"] Oct 30 13:11:48.925239 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:11:48.925209 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c75c816_7912_4460_b898_ad9da761027b.slice/crio-5267607be752c93a90253858d83a2ad888e9e748f892734a4a7fa60611b9150c WatchSource:0}: Error finding container 5267607be752c93a90253858d83a2ad888e9e748f892734a4a7fa60611b9150c: Status 404 returned error can't find the container with id 5267607be752c93a90253858d83a2ad888e9e748f892734a4a7fa60611b9150c Oct 30 13:11:48.926844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:48.926825 5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:11:48.976957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:48.976917 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" event={"ID":"0c75c816-7912-4460-b898-ad9da761027b","Type":"ContainerStarted","Data":"5267607be752c93a90253858d83a2ad888e9e748f892734a4a7fa60611b9150c"} Oct 30 13:11:49.039701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:49.039667 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:49.312955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:49.312849 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:49.985832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:49.985789 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" event={"ID":"0c75c816-7912-4460-b898-ad9da761027b","Type":"ContainerStarted","Data":"8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e"} Oct 30 13:11:49.986219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:49.985857 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:50.040466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:50.040410 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" podStartSLOduration=2.568014678 podStartE2EDuration="3.040394777s" totalImagesPullingTime="472.380099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:50.038377933 +0000 UTC m=+2328.165178832" watchObservedRunningTime="2026-10-30 13:11:50.040394777 +0000 UTC m=+2328.167195676" Oct 30 13:11:50.290390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:50.290287 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:51.022700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.022662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:51.361839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.361732 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:51.528513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.528474 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w"] Oct 30 13:11:51.533029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.533000 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:11:51.562923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.562893 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e9cb4a\"" Oct 30 13:11:51.569473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.569437 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w"] Oct 30 13:11:51.627034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.626950 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e9cb4a-dockercfg-dpjv6\"" Oct 30 13:11:51.658298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.657621 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e64645-cc47-496b-980b-f22d6ed896e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w\" (UID: \"60e64645-cc47-496b-980b-f22d6ed896e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:11:51.658298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.657863 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtmkj\" (UniqueName: \"kubernetes.io/projected/60e64645-cc47-496b-980b-f22d6ed896e7-kube-api-access-qtmkj\") pod \"payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w\" (UID: \"60e64645-cc47-496b-980b-f22d6ed896e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:11:51.689317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.688905 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:51.772412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.769632 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtmkj\" (UniqueName: \"kubernetes.io/projected/60e64645-cc47-496b-980b-f22d6ed896e7-kube-api-access-qtmkj\") pod \"payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w\" (UID: \"60e64645-cc47-496b-980b-f22d6ed896e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:11:51.772412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.769833 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e64645-cc47-496b-980b-f22d6ed896e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w\" (UID: \"60e64645-cc47-496b-980b-f22d6ed896e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:11:51.779286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.773204 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e64645-cc47-496b-980b-f22d6ed896e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w\" (UID: \"60e64645-cc47-496b-980b-f22d6ed896e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:11:51.866303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.857464 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtmkj\" (UniqueName: \"kubernetes.io/projected/60e64645-cc47-496b-980b-f22d6ed896e7-kube-api-access-qtmkj\") pod \"payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w\" (UID: \"60e64645-cc47-496b-980b-f22d6ed896e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:11:51.953429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:51.953396 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:52.149865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:52.149824 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:11:52.352287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:52.352227 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:52.460648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:52.460618 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w"] Oct 30 13:11:52.460792 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:11:52.460731 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e64645_cc47_496b_980b_f22d6ed896e7.slice/crio-26afd6712b533c57963b4ad2f3b019b40f1cd90afed932485078cac3c485e58b WatchSource:0}: Error finding container 26afd6712b533c57963b4ad2f3b019b40f1cd90afed932485078cac3c485e58b: Status 404 returned error can't find the container with id 26afd6712b533c57963b4ad2f3b019b40f1cd90afed932485078cac3c485e58b Oct 30 13:11:53.007685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:53.007648 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" event={"ID":"60e64645-cc47-496b-980b-f22d6ed896e7","Type":"ContainerStarted","Data":"36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc"} Oct 30 13:11:53.007889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:53.007717 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:11:53.007889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:53.007740 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" event={"ID":"60e64645-cc47-496b-980b-f22d6ed896e7","Type":"ContainerStarted","Data":"26afd6712b533c57963b4ad2f3b019b40f1cd90afed932485078cac3c485e58b"} Oct 30 13:11:53.060527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:53.060463 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" podStartSLOduration=2.060444611 podStartE2EDuration="2.060444611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:53.055723814 +0000 UTC m=+2331.182524711" watchObservedRunningTime="2026-10-30 13:11:53.060444611 +0000 UTC m=+2331.187245510" Oct 30 13:11:54.267124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:54.267083 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:54.598794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:54.598702 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:55.151150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.151110 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b"] Oct 30 13:11:55.151656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.151622 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" containerID="cri-o://1b51ab7a0ceba1e0f08ec20bd311d49760e16c415881c20a7d388b408b4a9107" gracePeriod=30 Oct 30 13:11:55.273163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.273128 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn"] Oct 30 13:11:55.280287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.280239 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.298996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.298955 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-517a25-openshift-default-dockercfg-dzh6f\"" Oct 30 13:11:55.300780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.300749 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn"] Oct 30 13:11:55.420277 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.420163 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:55.440309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.440225 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-credential-socket\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.440510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.440464 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-envoy\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.440572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.440533 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-data\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.440679 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.440658 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-istio-token\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.440759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.440701 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb79059-60be-46d7-a8d5-249714d32aab-istio-podinfo\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.440821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.440766 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-socket\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.440821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.440791 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb79059-60be-46d7-a8d5-249714d32aab-istiod-ca-cert\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.440950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.440934 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h72mx\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-kube-api-access-h72mx\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.440997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.440981 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-certs\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.547477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.547428 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-envoy\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.547687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.547530 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-data\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.547687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.547595 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-istio-token\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.547687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.547628 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb79059-60be-46d7-a8d5-249714d32aab-istio-podinfo\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.547687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.547663 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-socket\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.547934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.547687 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb79059-60be-46d7-a8d5-249714d32aab-istiod-ca-cert\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.547934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.547796 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h72mx\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-kube-api-access-h72mx\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.547934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.547827 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-certs\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.547934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.547871 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-credential-socket\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.548959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.548505 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-data\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.548959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.548542 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-socket\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.549500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.549473 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-certs\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.549918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.549898 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-credential-socket\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.550965 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.550616 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb79059-60be-46d7-a8d5-249714d32aab-istiod-ca-cert\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.562445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.562329 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-envoy\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.564564 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.564518 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb79059-60be-46d7-a8d5-249714d32aab-istio-podinfo\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.584548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.584518 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-istio-token\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.584707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.584609 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72mx\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-kube-api-access-h72mx\") pod \"e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.597526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.597483 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:55.698765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:55.698663 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.002827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:56.002797 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:11:56.137013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:56.136984 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn"] Oct 30 13:11:56.137336 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:11:56.137311 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb79059_60be_46d7_a8d5_249714d32aab.slice/crio-c7ba1d6ace25378a0d50c7c678ec48285f9771dc70f68052647c2e1ebf65bdb3 WatchSource:0}: Error finding container c7ba1d6ace25378a0d50c7c678ec48285f9771dc70f68052647c2e1ebf65bdb3: Status 404 returned error can't find the container with id c7ba1d6ace25378a0d50c7c678ec48285f9771dc70f68052647c2e1ebf65bdb3 Oct 30 13:11:56.139936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:56.139903 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:56.140001 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:56.139971 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:56.140053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:56.139999 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:11:56.184186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:56.184135 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.390173 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:56.390124 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:56.610423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:56.607850 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:57.091108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:57.091067 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" event={"ID":"deb79059-60be-46d7-a8d5-249714d32aab","Type":"ContainerStarted","Data":"b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334"} Oct 30 13:11:57.091472 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:57.091444 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" event={"ID":"deb79059-60be-46d7-a8d5-249714d32aab","Type":"ContainerStarted","Data":"c7ba1d6ace25378a0d50c7c678ec48285f9771dc70f68052647c2e1ebf65bdb3"} Oct 30 13:11:57.600706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:57.600669 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:57.633657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:57.633621 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:57.729295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:57.726344 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podStartSLOduration=2.7263206970000002 podStartE2EDuration="2.726320697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:57.18094921 +0000 UTC m=+2335.307750109" watchObservedRunningTime="2026-10-30 13:11:57.726320697 +0000 UTC m=+2335.853121598" Oct 30 13:11:57.805295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:57.801368 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 13:11:57.867503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:57.867373 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": context deadline exceeded" Oct 30 13:11:57.964294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:57.963757 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:58.145403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:58.145205 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:58.166049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:58.165680 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:11:58.689293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:58.688377 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:58.752294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:58.751525 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:58.757290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:58.754101 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:58.857292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:58.857199 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:58.880294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:58.857932 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 13:11:58.977348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:58.964143 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:59.179292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:58.982135 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 13:11:59.179292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.068891 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:59.179292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.173615 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:59.297948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.297822 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:11:59.523799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.523754 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk"] Oct 30 13:11:59.631346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.629079 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:11:59.657403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.657370 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-7a8b38\"" Oct 30 13:11:59.657759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.657735 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-7a8b38-dockercfg-mrsnm\"" Oct 30 13:11:59.697554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.697504 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:59.750925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.750900 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 13:11:59.762507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.757600 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:59.766424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.757647 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:11:59.775689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.775653 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 13:11:59.823296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.819723 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 13:12:00.143636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.819256 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2cbef4-f022-4196-afac-643eda0f7855-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk\" (UID: \"ea2cbef4-f022-4196-afac-643eda0f7855\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:12:00.143636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.824330 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4w8\" (UniqueName: \"kubernetes.io/projected/ea2cbef4-f022-4196-afac-643eda0f7855-kube-api-access-lc4w8\") pod \"payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk\" (UID: \"ea2cbef4-f022-4196-afac-643eda0f7855\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:12:00.143636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.860818 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:00.143636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:11:59.965356 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 13:12:00.143636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:00.061639 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk"] Oct 30 13:12:00.143636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:00.082784 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4w8\" (UniqueName: \"kubernetes.io/projected/ea2cbef4-f022-4196-afac-643eda0f7855-kube-api-access-lc4w8\") pod \"payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk\" (UID: \"ea2cbef4-f022-4196-afac-643eda0f7855\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:12:00.143636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:00.083055 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2cbef4-f022-4196-afac-643eda0f7855-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk\" (UID: \"ea2cbef4-f022-4196-afac-643eda0f7855\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:12:00.413462 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:00.370012 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 13:12:00.621160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:00.621117 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2cbef4-f022-4196-afac-643eda0f7855-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk\" (UID: \"ea2cbef4-f022-4196-afac-643eda0f7855\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:12:00.969358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:00.889682 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 13:12:01.088682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.075623 5515 patch_prober.go:29] interesting pod/thanos-querier-55b7b78d57-rlt5q container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.11:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:12:01.088682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.075699 5515 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" podUID="dbafdcd8-268f-4128-b17e-10d3d92e2227" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.11:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.088682 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:12:01.078851 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:12:01.171717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.169444 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:12:01.171717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.169511 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.171717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.169918 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 13:12:01.262411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.262324 5515 patch_prober.go:29] interesting pod/thanos-querier-55b7b78d57-rlt5q container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:12:01.346292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.282600 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-55b7b78d57-rlt5q" podUID="dbafdcd8-268f-4128-b17e-10d3d92e2227" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.11:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.346292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.303044 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h" podUID="6bac6b94-7947-41ce-aeb6-3daf8d972487" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.346292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.337884 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 13:12:01.346292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.338672 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 13:12:01.355077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.354435 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4w8\" (UniqueName: \"kubernetes.io/projected/ea2cbef4-f022-4196-afac-643eda0f7855-kube-api-access-lc4w8\") pod \"payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk\" (UID: \"ea2cbef4-f022-4196-afac-643eda0f7855\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:12:01.358851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.358810 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.364633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.364594 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pnkfq" podUID="b2ea78e5-dd16-43b8-a556-4fea83e22c64" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 13:12:01.368372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.368343 5515 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-s2gdv container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.13:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:12:01.368629 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.368604 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-s2gdv" podUID="983c8f5c-68cf-42a4-b931-4ceb2b7e1e77" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.13:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.396316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.378820 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:12:01.396316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.378886 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.396316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.378954 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2" podUID="ba929e02-50ce-44cf-af12-21d91c229651" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 30 13:12:01.457758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.457710 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:01.535721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.535630 5515 generic.go:356] "Generic (PLEG): container finished" podID="c64555a8-b762-489a-b00b-e952c6253a85" containerID="1b51ab7a0ceba1e0f08ec20bd311d49760e16c415881c20a7d388b408b4a9107" exitCode=0 Oct 30 13:12:01.584987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.584904 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:12:01.585293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.585236 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.656413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.635972 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7cq2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:12:01.656413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.636044 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.656413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.637709 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:12:01.657345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.656379 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" podUID="543202e9-3b15-46b2-9dad-b6d056c087b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 13:12:01.660975 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:12:01.660944 5515 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.597s" Oct 30 13:12:01.661153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.661139 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx"] Oct 30 13:12:01.667681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.667657 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 13:12:01.667910 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.667897 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 13:12:01.668019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.668009 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx"] Oct 30 13:12:01.668258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.668145 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 13:12:01.668668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.668646 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" event={"ID":"c64555a8-b762-489a-b00b-e952c6253a85","Type":"ContainerDied","Data":"1b51ab7a0ceba1e0f08ec20bd311d49760e16c415881c20a7d388b408b4a9107"} Oct 30 13:12:01.671936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.668012 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:01.691432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.690209 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc" Oct 30 13:12:01.694192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.693943 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:01.789368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.777141 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xms\" (UniqueName: \"kubernetes.io/projected/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-kube-api-access-b9xms\") pod \"payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx\" (UID: \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:01.789368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.777760 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx\" (UID: \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:01.842911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.834259 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 13:12:01.842911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.834562 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 13:12:01.842911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.834614 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 13:12:01.868657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.867486 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 13:12:01.883793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.881420 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx\" (UID: \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:01.883793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.881845 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xms\" (UniqueName: \"kubernetes.io/projected/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-kube-api-access-b9xms\") pod \"payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx\" (UID: \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:01.884247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.884206 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx\" (UID: \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:01.890083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.890049 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:12:01.915743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.911728 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 13:12:01.922601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:01.922539 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xms\" (UniqueName: \"kubernetes.io/projected/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-kube-api-access-b9xms\") pod \"payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx\" (UID: \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:02.060026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.059989 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 13:12:02.166918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.166882 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk"] Oct 30 13:12:02.167530 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:12:02.167506 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2cbef4_f022_4196_afac_643eda0f7855.slice/crio-799cf04d0b8fbfd3e3da2c245e45487ad5d11e5095cb3a6e7471e9c09b505ecc WatchSource:0}: Error finding container 799cf04d0b8fbfd3e3da2c245e45487ad5d11e5095cb3a6e7471e9c09b505ecc: Status 404 returned error can't find the container with id 799cf04d0b8fbfd3e3da2c245e45487ad5d11e5095cb3a6e7471e9c09b505ecc Oct 30 13:12:02.168774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.168752 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:02.190322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190292 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-socket\") pod \"c64555a8-b762-489a-b00b-e952c6253a85\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " Oct 30 13:12:02.190446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190420 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-envoy\") pod \"c64555a8-b762-489a-b00b-e952c6253a85\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " Oct 30 13:12:02.190513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190456 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-certs\") pod \"c64555a8-b762-489a-b00b-e952c6253a85\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " Oct 30 13:12:02.190513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190504 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-kube-api-access-47vts\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-kube-api-access-47vts\") pod \"c64555a8-b762-489a-b00b-e952c6253a85\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " Oct 30 13:12:02.190619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190534 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-socket" pod "c64555a8-b762-489a-b00b-e952c6253a85" (UID: "c64555a8-b762-489a-b00b-e952c6253a85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:02.190619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190572 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-istio-token\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-istio-token\") pod \"c64555a8-b762-489a-b00b-e952c6253a85\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " Oct 30 13:12:02.190723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190680 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64555a8-b762-489a-b00b-e952c6253a85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64555a8-b762-489a-b00b-e952c6253a85-istiod-ca-cert\") pod \"c64555a8-b762-489a-b00b-e952c6253a85\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " Oct 30 13:12:02.190787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190719 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-data\") pod \"c64555a8-b762-489a-b00b-e952c6253a85\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " Oct 30 13:12:02.190896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190853 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-credential-socket\") pod \"c64555a8-b762-489a-b00b-e952c6253a85\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " Oct 30 13:12:02.190979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.190963 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c64555a8-b762-489a-b00b-e952c6253a85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64555a8-b762-489a-b00b-e952c6253a85-istio-podinfo\") pod \"c64555a8-b762-489a-b00b-e952c6253a85\" (UID: \"c64555a8-b762-489a-b00b-e952c6253a85\") " Oct 30 13:12:02.191195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.191172 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-data" pod "c64555a8-b762-489a-b00b-e952c6253a85" (UID: "c64555a8-b762-489a-b00b-e952c6253a85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:02.191427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.191225 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-credential-socket" pod "c64555a8-b762-489a-b00b-e952c6253a85" (UID: "c64555a8-b762-489a-b00b-e952c6253a85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:02.191427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.191225 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64555a8-b762-489a-b00b-e952c6253a85-istiod-ca-cert" pod "c64555a8-b762-489a-b00b-e952c6253a85" (UID: "c64555a8-b762-489a-b00b-e952c6253a85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:12:02.191704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.191656 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-certs" pod "c64555a8-b762-489a-b00b-e952c6253a85" (UID: "c64555a8-b762-489a-b00b-e952c6253a85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:02.239449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.192506 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c64555a8-b762-489a-b00b-e952c6253a85-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.239449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.192539 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.239449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.192595 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.239449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.192612 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.239449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.228345 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c64555a8-b762-489a-b00b-e952c6253a85-istio-podinfo" pod "c64555a8-b762-489a-b00b-e952c6253a85" (UID: "c64555a8-b762-489a-b00b-e952c6253a85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:12:02.239449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.228498 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-istio-token" pod "c64555a8-b762-489a-b00b-e952c6253a85" (UID: "c64555a8-b762-489a-b00b-e952c6253a85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:02.239449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.228815 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-kube-api-access-47vts" pod "c64555a8-b762-489a-b00b-e952c6253a85" (UID: "c64555a8-b762-489a-b00b-e952c6253a85"). InnerVolumeSpecName "kube-api-access-47vts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:02.239449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.231499 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-envoy" pod "c64555a8-b762-489a-b00b-e952c6253a85" (UID: "c64555a8-b762-489a-b00b-e952c6253a85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:12:02.331399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.294053 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.331399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.294095 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c64555a8-b762-489a-b00b-e952c6253a85-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.331399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.294138 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47vts\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-kube-api-access-47vts\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.331399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.294155 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c64555a8-b762-489a-b00b-e952c6253a85-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.331399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.294170 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c64555a8-b762-489a-b00b-e952c6253a85-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:12:02.543799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.543762 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" event={"ID":"ea2cbef4-f022-4196-afac-643eda0f7855","Type":"ContainerStarted","Data":"ea78a51d9f07b96f4c3173e946a87d43bd5d8b59c5e543873b335fa5ddae7ded"} Oct 30 13:12:02.543997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.543859 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" event={"ID":"ea2cbef4-f022-4196-afac-643eda0f7855","Type":"ContainerStarted","Data":"799cf04d0b8fbfd3e3da2c245e45487ad5d11e5095cb3a6e7471e9c09b505ecc"} Oct 30 13:12:02.544066 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.543991 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:12:02.545452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.545427 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" Oct 30 13:12:02.545452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.545439 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b" event={"ID":"c64555a8-b762-489a-b00b-e952c6253a85","Type":"ContainerDied","Data":"1976206a2317d51a216154acb3b6ccf1ded12684236b6ab5e03eb12b9936aa50"} Oct 30 13:12:02.545638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.545489 5515 scope.go:122] "RemoveContainer" containerID="1b51ab7a0ceba1e0f08ec20bd311d49760e16c415881c20a7d388b408b4a9107" Oct 30 13:12:02.564287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.564235 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx"] Oct 30 13:12:02.738723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.738596 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:02.754046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.753864 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" podStartSLOduration=3.7538378740000002 podStartE2EDuration="3.753837874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:02.632659577 +0000 UTC m=+2340.759460476" watchObservedRunningTime="2026-10-30 13:12:02.753837874 +0000 UTC m=+2340.880638774" Oct 30 13:12:02.772005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.771920 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b"] Oct 30 13:12:02.806586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.806543 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-xr84b"] Oct 30 13:12:02.984057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:02.983947 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:03.554352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:03.554310 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" event={"ID":"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd","Type":"ContainerStarted","Data":"10c9716038d30923c0d8cfc46b1e964fb1c994ed7a823a3323e1a9242944479e"} Oct 30 13:12:03.554820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:03.554374 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" event={"ID":"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd","Type":"ContainerStarted","Data":"68bdb099800e89b39929f0320521ca75bde4d6e0b11343405e6f75e654ae2f0d"} Oct 30 13:12:03.554820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:03.554511 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:03.622857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:03.622726 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" podStartSLOduration=4.622704101 podStartE2EDuration="4.622704101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:03.617952982 +0000 UTC m=+2341.744753882" watchObservedRunningTime="2026-10-30 13:12:03.622704101 +0000 UTC m=+2341.749505000" Oct 30 13:12:04.025618 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:04.025581 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64555a8-b762-489a-b00b-e952c6253a85" path="/var/lib/kubelet/pods/c64555a8-b762-489a-b00b-e952c6253a85/volumes" Oct 30 13:12:04.025980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:04.025956 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:12:04.578525 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:04.576259 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:04.812306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:04.809092 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:06.215317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:06.209015 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:07.820619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:07.820575 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:09.584679 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:09.584645 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:10.263999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:10.263954 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:10.628322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:10.628217 5515 generic.go:356] "Generic (PLEG): container finished" podID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerID="b8097d89590850f890f2e2a320c0b126a706f1daf2b5a5428896aab65133e1aa" exitCode=0 Oct 30 13:12:10.628322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:10.628289 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" event={"ID":"1ff0adac-2f41-4528-ab90-ff59a93db902","Type":"ContainerDied","Data":"b8097d89590850f890f2e2a320c0b126a706f1daf2b5a5428896aab65133e1aa"} Oct 30 13:12:10.628862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:10.628751 5515 scope.go:122] "RemoveContainer" containerID="b8097d89590850f890f2e2a320c0b126a706f1daf2b5a5428896aab65133e1aa" Oct 30 13:12:10.630346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:10.630310 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:10.630449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:10.630403 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:10.630449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:10.630442 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:11.125321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:11.125256 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:11.346216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:11.346173 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:11.586212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:11.586170 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:11.637544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:11.637506 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" event={"ID":"1ff0adac-2f41-4528-ab90-ff59a93db902","Type":"ContainerStarted","Data":"7b8b2e929d2926ab8e63128f20ee52d8e2fc8e3b88fd4f3a4cfbb5441c8c99a0"} Oct 30 13:12:12.064729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.064534 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:12:12.065153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.064976 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:12:12.065374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.065349 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:12:12.427293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.422858 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:12.639104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.639064 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:12.949889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.949852 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6"] Oct 30 13:12:12.950543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.950526 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" Oct 30 13:12:12.950543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.950543 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" Oct 30 13:12:12.950659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.950644 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64555a8-b762-489a-b00b-e952c6253a85" containerName="istio-proxy" Oct 30 13:12:12.954370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.954349 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:12.981867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:12.981837 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6"] Oct 30 13:12:13.019633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.019599 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-7a8b38-serving-cert\"" Oct 30 13:12:13.040958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.040924 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.041155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.041128 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1973c8-eb1b-471e-a568-b81943e66597-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.041379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.041356 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.041527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.041500 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czlvw\" (UniqueName: \"kubernetes.io/projected/9a1973c8-eb1b-471e-a568-b81943e66597-kube-api-access-czlvw\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.061291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.061245 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:12:13.061459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.061312 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:12:13.121853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.121815 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.172292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.155512 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.172292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.155675 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czlvw\" (UniqueName: \"kubernetes.io/projected/9a1973c8-eb1b-471e-a568-b81943e66597-kube-api-access-czlvw\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.172292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.155756 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.172292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.155892 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1973c8-eb1b-471e-a568-b81943e66597-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.172292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.156993 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1973c8-eb1b-471e-a568-b81943e66597-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.172292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.167477 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.213550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.213472 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.242289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.237321 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czlvw\" (UniqueName: \"kubernetes.io/projected/9a1973c8-eb1b-471e-a568-b81943e66597-kube-api-access-czlvw\") pod \"maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.271704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.271664 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:13.326646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.326472 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:13.557564 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.557533 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:12:13.766379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:13.766349 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6"] Oct 30 13:12:13.766998 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:12:13.766963 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1973c8_eb1b_471e_a568_b81943e66597.slice/crio-933596e4ad9e00076c6da55c412e6d657f74b4760f20c82d95c79d78582286a3 WatchSource:0}: Error finding container 933596e4ad9e00076c6da55c412e6d657f74b4760f20c82d95c79d78582286a3: Status 404 returned error can't find the container with id 933596e4ad9e00076c6da55c412e6d657f74b4760f20c82d95c79d78582286a3 Oct 30 13:12:14.075061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:14.066931 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:12:14.075061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:14.067053 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:12:14.584711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:14.584681 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:12:14.673775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:14.673735 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" event={"ID":"9a1973c8-eb1b-471e-a568-b81943e66597","Type":"ContainerStarted","Data":"53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8"} Oct 30 13:12:14.674061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:14.674034 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" event={"ID":"9a1973c8-eb1b-471e-a568-b81943e66597","Type":"ContainerStarted","Data":"933596e4ad9e00076c6da55c412e6d657f74b4760f20c82d95c79d78582286a3"} Oct 30 13:12:14.675302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:14.675263 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:14.733018 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:14.732951 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" podStartSLOduration=2.23545261 podStartE2EDuration="2.732932888s" totalImagesPullingTime="497.480278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:14.729750834 +0000 UTC m=+2352.856551736" watchObservedRunningTime="2026-10-30 13:12:14.732932888 +0000 UTC m=+2352.859733785" Oct 30 13:12:15.067815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:15.067704 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:12:15.075338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:15.068484 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:12:15.075624 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:15.073660 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:15.815293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:15.813830 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:16.055905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:16.055869 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:16.061263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:16.061242 5515 patch_prober.go:29] interesting pod/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 13:12:16.061378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:16.061328 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 13:12:16.880020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:16.879989 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:17.117628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:17.117590 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:12:17.118556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:17.118531 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:12:17.119658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:17.119632 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:12:17.220449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:17.220419 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:17.716422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:17.716387 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:12:18.453683 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.453643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:18.682640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.682605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:18.707124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.707035 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m"] Oct 30 13:12:18.718090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.718062 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:18.738971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.738941 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ca097a-dockercfg-cn9mm\"" Oct 30 13:12:18.752822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.752696 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ca097a\"" Oct 30 13:12:18.766438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.766403 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m"] Oct 30 13:12:18.805488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.805449 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m\" (UID: \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:18.805696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.805581 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtdm\" (UniqueName: \"kubernetes.io/projected/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-kube-api-access-2vtdm\") pod \"payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m\" (UID: \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:18.907329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.907284 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtdm\" (UniqueName: \"kubernetes.io/projected/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-kube-api-access-2vtdm\") pod \"payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m\" (UID: \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:18.907575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.907558 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m\" (UID: \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:18.908252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.908233 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m\" (UID: \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:18.955005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:18.954970 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtdm\" (UniqueName: \"kubernetes.io/projected/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-kube-api-access-2vtdm\") pod \"payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m\" (UID: \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:19.036480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:19.036385 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:19.285451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:19.277133 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:19.475295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:19.470606 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m"] Oct 30 13:12:19.518772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:19.518738 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:19.739502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:19.739382 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:19.740851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:19.740818 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" event={"ID":"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2","Type":"ContainerStarted","Data":"d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4"} Oct 30 13:12:19.741026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:19.740891 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" event={"ID":"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2","Type":"ContainerStarted","Data":"11c534fd68e39668dcdd22fe06bfdec19043a2b37e3e7b9c7b6f3aca2ae38e71"} Oct 30 13:12:19.741026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:19.740926 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:21.507500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:21.507463 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:21.696961 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:21.696931 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:12:21.747456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:21.747397 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" podStartSLOduration=3.747381604 podStartE2EDuration="3.747381604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:19.785157565 +0000 UTC m=+2357.911958462" watchObservedRunningTime="2026-10-30 13:12:21.747381604 +0000 UTC m=+2359.874182501" Oct 30 13:12:23.246718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:23.246683 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:23.505846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:23.505756 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:23.827512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:23.827412 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:24.970692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:24.970655 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:24.997740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:24.997702 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj"] Oct 30 13:12:25.002145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.002127 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:25.028311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.028258 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj"] Oct 30 13:12:25.031962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.031932 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-bd2aaf\"" Oct 30 13:12:25.032223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.031987 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-bd2aaf-dockercfg-x6lrh\"" Oct 30 13:12:25.035365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.035327 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng"] Oct 30 13:12:25.039883 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.039856 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:25.062645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.062612 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng"] Oct 30 13:12:25.100134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.100099 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e0bbee-c23a-4e45-9754-b3c1375e66ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng\" (UID: \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:25.100342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.100261 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b299215-d93e-442f-aae4-87fee6eb584e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj\" (UID: \"5b299215-d93e-442f-aae4-87fee6eb584e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:25.100438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.100417 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48fd\" (UniqueName: \"kubernetes.io/projected/5b299215-d93e-442f-aae4-87fee6eb584e-kube-api-access-w48fd\") pod \"payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj\" (UID: \"5b299215-d93e-442f-aae4-87fee6eb584e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:25.100607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.100590 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjjv\" (UniqueName: \"kubernetes.io/projected/80e0bbee-c23a-4e45-9754-b3c1375e66ff-kube-api-access-gcjjv\") pod \"payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng\" (UID: \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:25.196553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.196518 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:25.202164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.202133 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48fd\" (UniqueName: \"kubernetes.io/projected/5b299215-d93e-442f-aae4-87fee6eb584e-kube-api-access-w48fd\") pod \"payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj\" (UID: \"5b299215-d93e-442f-aae4-87fee6eb584e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:25.202832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.202805 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjjv\" (UniqueName: \"kubernetes.io/projected/80e0bbee-c23a-4e45-9754-b3c1375e66ff-kube-api-access-gcjjv\") pod \"payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng\" (UID: \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:25.203449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.203427 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e0bbee-c23a-4e45-9754-b3c1375e66ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng\" (UID: \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:25.208498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.208468 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b299215-d93e-442f-aae4-87fee6eb584e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj\" (UID: \"5b299215-d93e-442f-aae4-87fee6eb584e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:25.224784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.224437 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b299215-d93e-442f-aae4-87fee6eb584e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj\" (UID: \"5b299215-d93e-442f-aae4-87fee6eb584e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:25.225213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.225189 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e0bbee-c23a-4e45-9754-b3c1375e66ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng\" (UID: \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:25.251304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.238937 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjjv\" (UniqueName: \"kubernetes.io/projected/80e0bbee-c23a-4e45-9754-b3c1375e66ff-kube-api-access-gcjjv\") pod \"payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng\" (UID: \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:25.258338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.258253 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48fd\" (UniqueName: \"kubernetes.io/projected/5b299215-d93e-442f-aae4-87fee6eb584e-kube-api-access-w48fd\") pod \"payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj\" (UID: \"5b299215-d93e-442f-aae4-87fee6eb584e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:25.321153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.321123 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:25.366014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.363486 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:25.662748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.661193 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:25.713706 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:12:25.713653 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b299215_d93e_442f_aae4_87fee6eb584e.slice/crio-cc92e33a5e9f4e27a09bc632043c95cae1488e3511d3b87c7be80cdf1f6f60d6 WatchSource:0}: Error finding container cc92e33a5e9f4e27a09bc632043c95cae1488e3511d3b87c7be80cdf1f6f60d6: Status 404 returned error can't find the container with id cc92e33a5e9f4e27a09bc632043c95cae1488e3511d3b87c7be80cdf1f6f60d6 Oct 30 13:12:25.715327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.715284 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj"] Oct 30 13:12:25.728069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.728037 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng"] Oct 30 13:12:25.732349 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:12:25.732317 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e0bbee_c23a_4e45_9754_b3c1375e66ff.slice/crio-f7fda4a83aedb47db67229124f16ecd49134a55cdb0113b9835b81e3aa12a341 WatchSource:0}: Error finding container f7fda4a83aedb47db67229124f16ecd49134a55cdb0113b9835b81e3aa12a341: Status 404 returned error can't find the container with id f7fda4a83aedb47db67229124f16ecd49134a55cdb0113b9835b81e3aa12a341 Oct 30 13:12:25.812080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.812040 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" event={"ID":"5b299215-d93e-442f-aae4-87fee6eb584e","Type":"ContainerStarted","Data":"30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451"} Oct 30 13:12:25.812256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.812098 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" event={"ID":"5b299215-d93e-442f-aae4-87fee6eb584e","Type":"ContainerStarted","Data":"cc92e33a5e9f4e27a09bc632043c95cae1488e3511d3b87c7be80cdf1f6f60d6"} Oct 30 13:12:25.812256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.812160 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:25.813741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.813718 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" event={"ID":"80e0bbee-c23a-4e45-9754-b3c1375e66ff","Type":"ContainerStarted","Data":"ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd"} Oct 30 13:12:25.813839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.813761 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:25.813839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.813778 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" event={"ID":"80e0bbee-c23a-4e45-9754-b3c1375e66ff","Type":"ContainerStarted","Data":"f7fda4a83aedb47db67229124f16ecd49134a55cdb0113b9835b81e3aa12a341"} Oct 30 13:12:25.854753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.854687 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" podStartSLOduration=1.854667939 podStartE2EDuration="1.854667939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:25.853319999 +0000 UTC m=+2363.980120898" watchObservedRunningTime="2026-10-30 13:12:25.854667939 +0000 UTC m=+2363.981468838" Oct 30 13:12:25.911225 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:25.911156 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" podStartSLOduration=0.911136732 podStartE2EDuration="911.136732ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:25.906227532 +0000 UTC m=+2364.033028443" watchObservedRunningTime="2026-10-30 13:12:25.911136732 +0000 UTC m=+2364.037937631" Oct 30 13:12:27.675292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:27.674492 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:28.989462 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:12:28.989424 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:12:29.296963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:29.296867 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:29.746295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:29.746246 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:30.234113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.234078 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:30.604498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.604417 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd"] Oct 30 13:12:30.610395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.610366 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.631730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.631693 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-92855c-openshift-default-dockercfg-ql7sc\"" Oct 30 13:12:30.647690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.647655 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd"] Oct 30 13:12:30.759293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.753710 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-certs\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.759293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.753848 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-istio-token\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.759293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.753952 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a66893-8ec4-4286-b2de-52c5202295de-istiod-ca-cert\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.784390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.759612 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-data\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.784390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.759751 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-credential-socket\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.784390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.759786 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ts6\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-kube-api-access-f6ts6\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.784390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.759858 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a66893-8ec4-4286-b2de-52c5202295de-istio-podinfo\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.784390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.759965 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-socket\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.784390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.760086 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-envoy\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.807291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.802437 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:30.825294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.821683 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:12:30.957097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.957053 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-certs\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.960879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.960844 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-istio-token\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.961323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.961301 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a66893-8ec4-4286-b2de-52c5202295de-istiod-ca-cert\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.968857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.964517 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-data\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.990998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.990959 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a66893-8ec4-4286-b2de-52c5202295de-istiod-ca-cert\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.991263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.991239 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-credential-socket\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.991462 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.991444 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ts6\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-kube-api-access-f6ts6\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.991613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.991595 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a66893-8ec4-4286-b2de-52c5202295de-istio-podinfo\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.991768 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.991725 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-credential-socket\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.991768 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.991728 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-socket\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.991901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.991826 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-envoy\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:30.992244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:30.992222 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-socket\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:31.002565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.000484 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a66893-8ec4-4286-b2de-52c5202295de-istio-podinfo\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:31.008286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.008240 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-data\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:31.008530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.008510 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-certs\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:31.009655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.009609 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-envoy\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:31.016712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.016687 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-istio-token\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:31.018012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.017987 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ts6\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-kube-api-access-f6ts6\") pod \"e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:31.199296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.198905 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:31.232174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.232087 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:31.411480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.411445 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:31.622570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.622398 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd"] Oct 30 13:12:31.657315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.654780 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:31.657315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.654892 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:31.657315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.654939 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:31.974883 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.974841 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" event={"ID":"e8a66893-8ec4-4286-b2de-52c5202295de","Type":"ContainerStarted","Data":"47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc"} Oct 30 13:12:31.975092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:31.974898 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" event={"ID":"e8a66893-8ec4-4286-b2de-52c5202295de","Type":"ContainerStarted","Data":"fcf9c0a8e0e98610e68dd757352eff08d049a285e9cb391657e9bfbd405a55ef"} Oct 30 13:12:32.042666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:32.042505 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" podStartSLOduration=2.042486679 podStartE2EDuration="2.042486679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:32.037895815 +0000 UTC m=+2370.164696714" watchObservedRunningTime="2026-10-30 13:12:32.042486679 +0000 UTC m=+2370.169287577" Oct 30 13:12:32.241346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:32.240005 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:32.360700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:32.352599 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:32.982346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:32.982312 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:32.983568 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:32.983539 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:12:33.488306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:33.487366 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:34.720083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:34.720044 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:35.034394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.034305 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:35.682840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.682810 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph"] Oct 30 13:12:35.687735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.687709 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.735159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.735128 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph"] Oct 30 13:12:35.759213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.759177 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-metrics-certs\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.759592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.759565 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aaf89b-5970-46e1-b575-bc01f7b96862-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.759871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.759850 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8tb\" (UniqueName: \"kubernetes.io/projected/18aaf89b-5970-46e1-b575-bc01f7b96862-kube-api-access-tf8tb\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.760044 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.760014 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.801645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.801608 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ca097a-serving-cert\"" Oct 30 13:12:35.861670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.861641 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8tb\" (UniqueName: \"kubernetes.io/projected/18aaf89b-5970-46e1-b575-bc01f7b96862-kube-api-access-tf8tb\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.861892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.861691 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.861892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.861800 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-metrics-certs\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.861892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.861854 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aaf89b-5970-46e1-b575-bc01f7b96862-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.861892 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:12:35.861869 5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-serving-cert: secret "maas-api-e2e-praxis-b-ca097a-serving-cert" not found Oct 30 13:12:35.862111 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:12:35.861953 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls podName:18aaf89b-5970-46e1-b575-bc01f7b96862 nodeName:}" failed. No retries permitted until 2026-10-30 13:12:36.361930321 +0000 UTC m=+2374.488731196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls") pod "maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" (UID: "18aaf89b-5970-46e1-b575-bc01f7b96862") : secret "maas-api-e2e-praxis-b-ca097a-serving-cert" not found Oct 30 13:12:35.862824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.862796 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aaf89b-5970-46e1-b575-bc01f7b96862-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.865167 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.865143 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-metrics-certs\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:35.900704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.900668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:35.921136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:35.921101 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8tb\" (UniqueName: \"kubernetes.io/projected/18aaf89b-5970-46e1-b575-bc01f7b96862-kube-api-access-tf8tb\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:36.125232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:36.125149 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:36.373753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:36.373717 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:36.376358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:36.376307 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:36.637303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:36.635922 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:36.871782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:36.863831 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:36.875128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:36.875100 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:12:36.886002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:36.884993 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:12:37.025447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:37.025419 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph"] Oct 30 13:12:37.051963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:37.051922 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" event={"ID":"18aaf89b-5970-46e1-b575-bc01f7b96862","Type":"ContainerStarted","Data":"e38a9aa375d84810c14591a5d6440524abcc9e527da4233ab67db8bcc2e4baa1"} Oct 30 13:12:37.313355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:37.313077 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:38.085330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:38.055169 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:38.359471 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:38.358251 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:38.390031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:38.389984 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" event={"ID":"18aaf89b-5970-46e1-b575-bc01f7b96862","Type":"ContainerStarted","Data":"4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8"} Oct 30 13:12:38.390200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:38.390156 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:38.428041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:38.427978 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" podStartSLOduration=3.102723359 podStartE2EDuration="3.427959996s" totalImagesPullingTime="325.236637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:38.416285075 +0000 UTC m=+2376.543085971" watchObservedRunningTime="2026-10-30 13:12:38.427959996 +0000 UTC m=+2376.554760898" Oct 30 13:12:38.568920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:38.568255 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:39.095712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:39.095680 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:39.356968 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:39.356826 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:40.267147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:40.267082 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:41.980381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:41.975783 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:42.303859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:42.303771 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:42.577248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:42.577165 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:43.579892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:43.579233 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": read tcp 10.134.0.2:40166->10.134.0.159:15021: read: connection reset by peer" start-of-body= Oct 30 13:12:43.579892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:43.579316 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": read tcp 10.134.0.2:40166->10.134.0.159:15021: read: connection reset by peer" Oct 30 13:12:43.962781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:43.962750 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:44.207399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.193486 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq"] Oct 30 13:12:44.346916 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.346738 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:44.369042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.368873 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq"] Oct 30 13:12:44.369296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.369079 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:44.399713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.399685 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-98dabb\"" Oct 30 13:12:44.399873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.399766 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-98dabb-dockercfg-964c5\"" Oct 30 13:12:44.435664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.435636 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:12:44.442113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.442079 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn8cl\" (UniqueName: \"kubernetes.io/projected/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-kube-api-access-nn8cl\") pod \"payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq\" (UID: \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:44.442242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.442192 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq\" (UID: \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:44.454999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.454962 5515 generic.go:356] "Generic (PLEG): container finished" podID="deb79059-60be-46d7-a8d5-249714d32aab" containerID="b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334" exitCode=0 Oct 30 13:12:44.455183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.455037 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" event={"ID":"deb79059-60be-46d7-a8d5-249714d32aab","Type":"ContainerDied","Data":"b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334"} Oct 30 13:12:44.455562 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.455546 5515 scope.go:122] "RemoveContainer" containerID="b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334" Oct 30 13:12:44.457432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.457403 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:44.457512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.457493 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:44.457555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.457544 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:12:44.543709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.543668 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq\" (UID: \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:44.544138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.544116 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn8cl\" (UniqueName: \"kubernetes.io/projected/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-kube-api-access-nn8cl\") pod \"payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq\" (UID: \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:44.546394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.546355 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq\" (UID: \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:44.573752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.573723 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn8cl\" (UniqueName: \"kubernetes.io/projected/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-kube-api-access-nn8cl\") pod \"payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq\" (UID: \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:44.695924 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:44.695890 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:45.006995 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:12:45.006963 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2ea174_b297_44ab_96ea_ee1eaf59dbde.slice/crio-25566b277bab95c867b0c4aef2d24403e73d617adf412823b54024b239593b64 WatchSource:0}: Error finding container 25566b277bab95c867b0c4aef2d24403e73d617adf412823b54024b239593b64: Status 404 returned error can't find the container with id 25566b277bab95c867b0c4aef2d24403e73d617adf412823b54024b239593b64 Oct 30 13:12:45.007221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.007196 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq"] Oct 30 13:12:45.464021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.463975 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" event={"ID":"ca2ea174-b297-44ab-96ea-ee1eaf59dbde","Type":"ContainerStarted","Data":"6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707"} Oct 30 13:12:45.464215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.464042 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" event={"ID":"ca2ea174-b297-44ab-96ea-ee1eaf59dbde","Type":"ContainerStarted","Data":"25566b277bab95c867b0c4aef2d24403e73d617adf412823b54024b239593b64"} Oct 30 13:12:45.464215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.464069 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:45.465966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.465939 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" event={"ID":"deb79059-60be-46d7-a8d5-249714d32aab","Type":"ContainerStarted","Data":"a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976"} Oct 30 13:12:45.491148 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.491118 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg"] Oct 30 13:12:45.516290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.515935 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" podStartSLOduration=1.515916556 podStartE2EDuration="1.515916556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:45.512914851 +0000 UTC m=+2383.639715750" watchObservedRunningTime="2026-10-30 13:12:45.515916556 +0000 UTC m=+2383.642717455" Oct 30 13:12:45.521253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.519088 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.529791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.529759 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg"] Oct 30 13:12:45.536491 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.536461 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-bd2aaf-serving-cert\"" Oct 30 13:12:45.571100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.571061 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.571339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.571145 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.571635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.571614 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.571701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.571659 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f787z\" (UniqueName: \"kubernetes.io/projected/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-kube-api-access-f787z\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.618725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.618598 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:12:45.618952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.618826 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 13:12:45.618952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.618891 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 13:12:45.678379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.678348 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.678575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.678391 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f787z\" (UniqueName: \"kubernetes.io/projected/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-kube-api-access-f787z\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.678575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.678515 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.678701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.678587 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.678701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.678656 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:45.679348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.679322 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.681174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.681146 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.681970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.681943 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.730028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.729955 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f787z\" (UniqueName: \"kubernetes.io/projected/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-kube-api-access-f787z\") pod \"maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.842085 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.842054 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:45.948291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:45.948182 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:46.358346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:46.358172 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:46.375935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:46.375899 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg"] Oct 30 13:12:46.390776 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:12:46.390736 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b8ed75_2249_4244_bdfd_29cfa1bfd7dc.slice/crio-ab5f8855e6340744fa42cfa2803a7a774fe79ef25ebefb9fb022d9e79592be8a WatchSource:0}: Error finding container ab5f8855e6340744fa42cfa2803a7a774fe79ef25ebefb9fb022d9e79592be8a: Status 404 returned error can't find the container with id ab5f8855e6340744fa42cfa2803a7a774fe79ef25ebefb9fb022d9e79592be8a Oct 30 13:12:46.473692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:46.473654 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" event={"ID":"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc","Type":"ContainerStarted","Data":"ab5f8855e6340744fa42cfa2803a7a774fe79ef25ebefb9fb022d9e79592be8a"} Oct 30 13:12:46.617348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:46.616743 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 13:12:46.617348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:46.616808 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 13:12:46.863529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:46.863492 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.097730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:47.097673 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.483160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:47.483126 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" event={"ID":"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc","Type":"ContainerStarted","Data":"d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1"} Oct 30 13:12:47.483351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:47.483202 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:47.530618 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:47.530565 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" podStartSLOduration=2.20544204 podStartE2EDuration="2.530550775s" totalImagesPullingTime="325.108735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:47.527600905 +0000 UTC m=+2385.654401805" watchObservedRunningTime="2026-10-30 13:12:47.530550775 +0000 UTC m=+2385.657351672" Oct 30 13:12:47.613294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:47.605294 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:47.613294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:47.605466 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 13:12:47.613294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:47.605511 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 13:12:47.712293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:47.702926 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:12:48.603292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:48.599623 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 13:12:48.603292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:48.599682 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 13:12:49.604288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:49.599024 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 13:12:49.604288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:49.599093 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 13:12:49.626254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:49.626212 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:50.602855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:50.601696 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 13:12:50.602855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:50.601760 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 13:12:51.016342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:51.016309 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.275781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:51.275688 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.625013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:51.624917 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:12:51.626572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:51.626543 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:12:51.645752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:51.645527 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:51.645961 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:51.645905 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:12:52.829560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.829523 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4"] Oct 30 13:12:52.834524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.834501 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:12:52.862394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.862359 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4"] Oct 30 13:12:52.873301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.872041 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d828ed9-4e9f-4194-a442-bd8f9059be68-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4\" (UID: \"3d828ed9-4e9f-4194-a442-bd8f9059be68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:12:52.873301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.872451 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mzm\" (UniqueName: \"kubernetes.io/projected/3d828ed9-4e9f-4194-a442-bd8f9059be68-kube-api-access-78mzm\") pod \"payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4\" (UID: \"3d828ed9-4e9f-4194-a442-bd8f9059be68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:12:52.894032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.893977 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:52.966308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.962628 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-241ffa-dockercfg-shrph\"" Oct 30 13:12:52.966308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.962732 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-241ffa\"" Oct 30 13:12:52.994291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.983566 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d828ed9-4e9f-4194-a442-bd8f9059be68-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4\" (UID: \"3d828ed9-4e9f-4194-a442-bd8f9059be68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:12:52.994291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.983822 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mzm\" (UniqueName: \"kubernetes.io/projected/3d828ed9-4e9f-4194-a442-bd8f9059be68-kube-api-access-78mzm\") pod \"payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4\" (UID: \"3d828ed9-4e9f-4194-a442-bd8f9059be68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:12:52.994291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:52.985304 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d828ed9-4e9f-4194-a442-bd8f9059be68-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4\" (UID: \"3d828ed9-4e9f-4194-a442-bd8f9059be68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:12:53.028290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:53.027965 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mzm\" (UniqueName: \"kubernetes.io/projected/3d828ed9-4e9f-4194-a442-bd8f9059be68-kube-api-access-78mzm\") pod \"payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4\" (UID: \"3d828ed9-4e9f-4194-a442-bd8f9059be68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:12:53.155378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:53.153617 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:12:53.160657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:53.160618 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:53.482956 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:12:53.482923 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d828ed9_4e9f_4194_a442_bd8f9059be68.slice/crio-17046ee6abd06af140d8730882b7a954821b5ef909e4f93fd756dff6788b29ab WatchSource:0}: Error finding container 17046ee6abd06af140d8730882b7a954821b5ef909e4f93fd756dff6788b29ab: Status 404 returned error can't find the container with id 17046ee6abd06af140d8730882b7a954821b5ef909e4f93fd756dff6788b29ab Oct 30 13:12:53.483896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:53.483875 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4"] Oct 30 13:12:53.524720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:53.524693 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:12:53.625599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:53.623398 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" event={"ID":"3d828ed9-4e9f-4194-a442-bd8f9059be68","Type":"ContainerStarted","Data":"17046ee6abd06af140d8730882b7a954821b5ef909e4f93fd756dff6788b29ab"} Oct 30 13:12:53.657839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:53.657808 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:54.630753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:54.630716 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" event={"ID":"3d828ed9-4e9f-4194-a442-bd8f9059be68","Type":"ContainerStarted","Data":"22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb"} Oct 30 13:12:54.631228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:54.630843 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:12:54.676338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:54.676278 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" podStartSLOduration=2.676251137 podStartE2EDuration="2.676251137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:54.673416221 +0000 UTC m=+2392.800217120" watchObservedRunningTime="2026-10-30 13:12:54.676251137 +0000 UTC m=+2392.803052042" Oct 30 13:12:54.849299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:54.849229 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w"] Oct 30 13:12:54.853295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:54.849954 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" containerID="cri-o://39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c" gracePeriod=30 Oct 30 13:12:55.406410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:55.406256 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:56.477180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:56.477144 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:12:56.899689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:56.899653 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:57.112423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:57.112386 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:57.318644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:57.318554 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:58.655234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:58.655194 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:12:58.990757 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:12:58.990672 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:12:59.821803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:59.821764 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-t696w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:12:59.822227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:12:59.821853 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:13:00.137763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.137735 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.237448 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-data\") pod \"1f6a2964-91e7-4566-8121-59a460c1aeb7\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.237495 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-credential-socket\") pod \"1f6a2964-91e7-4566-8121-59a460c1aeb7\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.237551 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-certs\") pod \"1f6a2964-91e7-4566-8121-59a460c1aeb7\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.237610 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6a2964-91e7-4566-8121-59a460c1aeb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a2964-91e7-4566-8121-59a460c1aeb7-istiod-ca-cert\") pod \"1f6a2964-91e7-4566-8121-59a460c1aeb7\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.237651 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-token\") pod \"1f6a2964-91e7-4566-8121-59a460c1aeb7\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.237696 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-kube-api-access-q8z56\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-kube-api-access-q8z56\") pod \"1f6a2964-91e7-4566-8121-59a460c1aeb7\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.237747 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-podinfo\") pod \"1f6a2964-91e7-4566-8121-59a460c1aeb7\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.237940 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-socket\") pod \"1f6a2964-91e7-4566-8121-59a460c1aeb7\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.238035 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-envoy\") pod \"1f6a2964-91e7-4566-8121-59a460c1aeb7\" (UID: \"1f6a2964-91e7-4566-8121-59a460c1aeb7\") " Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.247713 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6a2964-91e7-4566-8121-59a460c1aeb7-istiod-ca-cert" pod "1f6a2964-91e7-4566-8121-59a460c1aeb7" (UID: "1f6a2964-91e7-4566-8121-59a460c1aeb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.248059 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-data" pod "1f6a2964-91e7-4566-8121-59a460c1aeb7" (UID: "1f6a2964-91e7-4566-8121-59a460c1aeb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.248318 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-credential-socket" pod "1f6a2964-91e7-4566-8121-59a460c1aeb7" (UID: "1f6a2964-91e7-4566-8121-59a460c1aeb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.248548 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-certs" pod "1f6a2964-91e7-4566-8121-59a460c1aeb7" (UID: "1f6a2964-91e7-4566-8121-59a460c1aeb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.253017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.251666 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-kube-api-access-q8z56" pod "1f6a2964-91e7-4566-8121-59a460c1aeb7" (UID: "1f6a2964-91e7-4566-8121-59a460c1aeb7"). InnerVolumeSpecName "kube-api-access-q8z56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.254499 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.254460 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-socket" pod "1f6a2964-91e7-4566-8121-59a460c1aeb7" (UID: "1f6a2964-91e7-4566-8121-59a460c1aeb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.264068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.263984 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-podinfo" pod "1f6a2964-91e7-4566-8121-59a460c1aeb7" (UID: "1f6a2964-91e7-4566-8121-59a460c1aeb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:13:00.265839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.265617 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-token" pod "1f6a2964-91e7-4566-8121-59a460c1aeb7" (UID: "1f6a2964-91e7-4566-8121-59a460c1aeb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:13:00.290199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.285879 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-envoy" pod "1f6a2964-91e7-4566-8121-59a460c1aeb7" (UID: "1f6a2964-91e7-4566-8121-59a460c1aeb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:13:00.340080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.340044 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6a2964-91e7-4566-8121-59a460c1aeb7-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.340080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.340079 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.340421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.340091 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8z56\" (UniqueName: \"kubernetes.io/projected/1f6a2964-91e7-4566-8121-59a460c1aeb7-kube-api-access-q8z56\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.340421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.340105 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.340421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.340117 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.340421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.340128 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.340421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.340142 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.340421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.340154 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.340421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.340166 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6a2964-91e7-4566-8121-59a460c1aeb7-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:13:00.389719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.389679 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:00.680750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.680667 5515 generic.go:356] "Generic (PLEG): container finished" podID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerID="39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c" exitCode=0 Oct 30 13:13:00.680750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.680714 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" Oct 30 13:13:00.680979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.680720 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" event={"ID":"1f6a2964-91e7-4566-8121-59a460c1aeb7","Type":"ContainerDied","Data":"39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c"} Oct 30 13:13:00.680979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.680841 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w" event={"ID":"1f6a2964-91e7-4566-8121-59a460c1aeb7","Type":"ContainerDied","Data":"639beab3f171fdcd0a161d4f8841f9ed70f9b8ef8fa2fe9b845632efd8dcf4cf"} Oct 30 13:13:00.680979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.680868 5515 scope.go:122] "RemoveContainer" containerID="39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c" Oct 30 13:13:00.691393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.691373 5515 scope.go:122] "RemoveContainer" containerID="39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c" Oct 30 13:13:00.691861 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:13:00.691836 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c\": container with ID starting with 39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c not found: ID does not exist" containerID="39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c" Oct 30 13:13:00.691914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.691872 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c"} err="failed to get container status \"39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c\": rpc error: code = NotFound desc = could not find container \"39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c\": container with ID starting with 39e5f5d922582db71aa55b133835ef321faedf6926fbbb4644b349fc014d361c not found: ID does not exist" Oct 30 13:13:00.768592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.768552 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w"] Oct 30 13:13:00.780374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:00.780341 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-t696w"] Oct 30 13:13:02.002529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:02.002498 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" path="/var/lib/kubelet/pods/1f6a2964-91e7-4566-8121-59a460c1aeb7/volumes" Oct 30 13:13:02.290035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:02.289930 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:03.347089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:03.347058 5515 scope.go:122] "RemoveContainer" containerID="53cf18f865c499d68a6ce5abd452202d44560f4ca4add20c5a73fa77b3fb1c5d" Oct 30 13:13:04.043943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:04.043903 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:05.436915 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.436874 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq"] Oct 30 13:13:05.437710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.437691 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" Oct 30 13:13:05.437782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.437712 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" Oct 30 13:13:05.437821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.437812 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6a2964-91e7-4566-8121-59a460c1aeb7" containerName="istio-proxy" Oct 30 13:13:05.441794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.441769 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.460165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.460130 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-98dabb-serving-cert\"" Oct 30 13:13:05.472389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.472352 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.472602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.472544 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d1adc-43f1-4344-97a4-618a8b5ea618-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.472715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.472660 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-575lb\" (UniqueName: \"kubernetes.io/projected/080d1adc-43f1-4344-97a4-618a8b5ea618-kube-api-access-575lb\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.472783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.472730 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.477073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.477043 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq"] Oct 30 13:13:05.577848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.577806 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-575lb\" (UniqueName: \"kubernetes.io/projected/080d1adc-43f1-4344-97a4-618a8b5ea618-kube-api-access-575lb\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.578050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.577893 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.578050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.577965 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.578149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.578125 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d1adc-43f1-4344-97a4-618a8b5ea618-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.578876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.578837 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d1adc-43f1-4344-97a4-618a8b5ea618-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.580739 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.580699 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.580859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.580717 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.602017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.601981 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-575lb\" (UniqueName: \"kubernetes.io/projected/080d1adc-43f1-4344-97a4-618a8b5ea618-kube-api-access-575lb\") pod \"maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.614799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.614760 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:05.683820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.682414 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:13:05.757178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.757087 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:05.831455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:05.831263 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:06.077263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:06.077221 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq"] Oct 30 13:13:06.078618 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:13:06.078591 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d1adc_43f1_4344_97a4_618a8b5ea618.slice/crio-16a5ee2232a238bcea8f5d71235b2932e421fa8cf1961c46d07993742d2d87b0 WatchSource:0}: Error finding container 16a5ee2232a238bcea8f5d71235b2932e421fa8cf1961c46d07993742d2d87b0: Status 404 returned error can't find the container with id 16a5ee2232a238bcea8f5d71235b2932e421fa8cf1961c46d07993742d2d87b0 Oct 30 13:13:06.778257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:06.775053 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:06.782139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:06.782098 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" event={"ID":"080d1adc-43f1-4344-97a4-618a8b5ea618","Type":"ContainerStarted","Data":"abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08"} Oct 30 13:13:06.782327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:06.782169 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" event={"ID":"080d1adc-43f1-4344-97a4-618a8b5ea618","Type":"ContainerStarted","Data":"16a5ee2232a238bcea8f5d71235b2932e421fa8cf1961c46d07993742d2d87b0"} Oct 30 13:13:06.782327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:06.782214 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:06.839166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:06.839111 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" podStartSLOduration=1.537166109 podStartE2EDuration="1.839095464s" totalImagesPullingTime="301.929355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:06.832607765 +0000 UTC m=+2404.959408657" watchObservedRunningTime="2026-10-30 13:13:06.839095464 +0000 UTC m=+2404.965896390" Oct 30 13:13:07.302307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:07.300744 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:08.917087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:08.917053 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:09.124937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:09.124893 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:10.607722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:10.607671 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:11.275047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:11.275008 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:13:11.278369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:11.278324 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:12.159781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:12.159692 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:12.798486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:12.798458 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:13:14.168020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:14.167917 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:14.386430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:14.386379 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:14.673655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:14.673620 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:15.186303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.183534 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:15.413417 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.413361 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:15.791375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.791342 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl"] Oct 30 13:13:15.818391 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.815615 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:15.825620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.825588 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl"] Oct 30 13:13:15.836026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.835987 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-241ffa-serving-cert\"" Oct 30 13:13:15.924869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.924829 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:15.925054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.924885 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:15.925149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.925116 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vh7b\" (UniqueName: \"kubernetes.io/projected/3282aba7-a1bd-498e-9013-f18c53c15574-kube-api-access-7vh7b\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:15.925192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.925177 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3282aba7-a1bd-498e-9013-f18c53c15574-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:15.990385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:15.990347 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.032309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.030735 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vh7b\" (UniqueName: \"kubernetes.io/projected/3282aba7-a1bd-498e-9013-f18c53c15574-kube-api-access-7vh7b\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:16.032309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.030841 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3282aba7-a1bd-498e-9013-f18c53c15574-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:16.032309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.030965 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:16.032309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.031025 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:16.051809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.051733 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3282aba7-a1bd-498e-9013-f18c53c15574-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:16.052137 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:13:16.052113 5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-serving-cert: secret "maas-api-e2e-shared-b-w0-241ffa-serving-cert" not found Oct 30 13:13:16.052358 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:13:16.052344 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls podName:3282aba7-a1bd-498e-9013-f18c53c15574 nodeName:}" failed. No retries permitted until 2026-10-30 13:13:16.552320462 +0000 UTC m=+2414.679121364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls") pod "maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" (UID: "3282aba7-a1bd-498e-9013-f18c53c15574") : secret "maas-api-e2e-shared-b-w0-241ffa-serving-cert" not found Oct 30 13:13:16.054337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.053153 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:16.093061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.093027 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vh7b\" (UniqueName: \"kubernetes.io/projected/3282aba7-a1bd-498e-9013-f18c53c15574-kube-api-access-7vh7b\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:16.223063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.223028 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.451236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.451202 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:16.652311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.646331 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:16.670363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.670322 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:16.743632 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:16.743530 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:17.371864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:17.371828 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl"] Oct 30 13:13:17.376084 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:13:17.376045 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3282aba7_a1bd_498e_9013_f18c53c15574.slice/crio-50dd34fd2c857a37f6ef22d0d2899cd603a48f60b90fa0eb558fa1d11ea119f7 WatchSource:0}: Error finding container 50dd34fd2c857a37f6ef22d0d2899cd603a48f60b90fa0eb558fa1d11ea119f7: Status 404 returned error can't find the container with id 50dd34fd2c857a37f6ef22d0d2899cd603a48f60b90fa0eb558fa1d11ea119f7 Oct 30 13:13:17.812409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:17.812248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:18.168357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:18.168320 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" event={"ID":"3282aba7-a1bd-498e-9013-f18c53c15574","Type":"ContainerStarted","Data":"84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27"} Oct 30 13:13:18.168570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:18.168382 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" event={"ID":"3282aba7-a1bd-498e-9013-f18c53c15574","Type":"ContainerStarted","Data":"50dd34fd2c857a37f6ef22d0d2899cd603a48f60b90fa0eb558fa1d11ea119f7"} Oct 30 13:13:18.168570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:18.168406 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:18.220940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:18.220882 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" podStartSLOduration=2.841558569 podStartE2EDuration="3.220867118s" totalImagesPullingTime="379.308549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:18.219121685 +0000 UTC m=+2416.345922596" watchObservedRunningTime="2026-10-30 13:13:18.220867118 +0000 UTC m=+2416.347668016" Oct 30 13:13:19.184823 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:19.184790 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:19.418918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:19.418884 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:19.661003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:19.657197 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:20.952624 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:20.952593 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.123917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.123856 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75"] Oct 30 13:13:21.155292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.145608 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:21.157923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.157876 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75"] Oct 30 13:13:21.193749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.193716 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-71fbeb\"" Oct 30 13:13:21.240365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.196105 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.305293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.215632 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-71fbeb-dockercfg-6tjdg\"" Oct 30 13:13:21.305293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.217041 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmlm\" (UniqueName: \"kubernetes.io/projected/3ef91a41-6516-4710-ae02-3f9dbeac790c-kube-api-access-zmmlm\") pod \"payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75\" (UID: \"3ef91a41-6516-4710-ae02-3f9dbeac790c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:21.305293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.300241 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef91a41-6516-4710-ae02-3f9dbeac790c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75\" (UID: \"3ef91a41-6516-4710-ae02-3f9dbeac790c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:21.401762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.401724 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmlm\" (UniqueName: \"kubernetes.io/projected/3ef91a41-6516-4710-ae02-3f9dbeac790c-kube-api-access-zmmlm\") pod \"payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75\" (UID: \"3ef91a41-6516-4710-ae02-3f9dbeac790c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:21.401958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.401771 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef91a41-6516-4710-ae02-3f9dbeac790c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75\" (UID: \"3ef91a41-6516-4710-ae02-3f9dbeac790c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:21.402617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.402589 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef91a41-6516-4710-ae02-3f9dbeac790c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75\" (UID: \"3ef91a41-6516-4710-ae02-3f9dbeac790c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:21.426195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.426160 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmlm\" (UniqueName: \"kubernetes.io/projected/3ef91a41-6516-4710-ae02-3f9dbeac790c-kube-api-access-zmmlm\") pod \"payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75\" (UID: \"3ef91a41-6516-4710-ae02-3f9dbeac790c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:21.590071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.589980 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:21.769552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.769469 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:21.899838 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:21.899803 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75"] Oct 30 13:13:21.900553 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:13:21.900520 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef91a41_6516_4710_ae02_3f9dbeac790c.slice/crio-f640612ac444c19c2fded87dc130e6f01c0413c6a30e681fa0606d8aaf20f889 WatchSource:0}: Error finding container f640612ac444c19c2fded87dc130e6f01c0413c6a30e681fa0606d8aaf20f889: Status 404 returned error can't find the container with id f640612ac444c19c2fded87dc130e6f01c0413c6a30e681fa0606d8aaf20f889 Oct 30 13:13:22.309177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:22.309040 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" event={"ID":"3ef91a41-6516-4710-ae02-3f9dbeac790c","Type":"ContainerStarted","Data":"1b75ebd186e8a7fc4222563c1f0c116a05a75ce3de01aed093ba008278c7a0d0"} Oct 30 13:13:22.309177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:22.309100 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:22.309177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:22.309119 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" event={"ID":"3ef91a41-6516-4710-ae02-3f9dbeac790c","Type":"ContainerStarted","Data":"f640612ac444c19c2fded87dc130e6f01c0413c6a30e681fa0606d8aaf20f889"} Oct 30 13:13:22.357411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:22.357194 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" podStartSLOduration=1.357177474 podStartE2EDuration="1.357177474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:22.353349558 +0000 UTC m=+2420.480150456" watchObservedRunningTime="2026-10-30 13:13:22.357177474 +0000 UTC m=+2420.483978372" Oct 30 13:13:23.609070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:23.608978 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:24.182011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:24.181977 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:13:24.326586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:24.326552 5515 generic.go:356] "Generic (PLEG): container finished" podID="e8a66893-8ec4-4286-b2de-52c5202295de" containerID="47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc" exitCode=0 Oct 30 13:13:24.326753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:24.326619 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" event={"ID":"e8a66893-8ec4-4286-b2de-52c5202295de","Type":"ContainerDied","Data":"47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc"} Oct 30 13:13:24.327076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:24.327055 5515 scope.go:122] "RemoveContainer" containerID="47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc" Oct 30 13:13:24.328758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:24.328728 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:24.328831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:24.328799 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:24.328874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:24.328832 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:13:24.903932 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:24.903892 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:25.110304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:25.109958 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:25.335776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:25.335687 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" event={"ID":"e8a66893-8ec4-4286-b2de-52c5202295de","Type":"ContainerStarted","Data":"13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d"} Oct 30 13:13:25.601208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:25.601103 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:26.233070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:26.233034 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:13:26.233554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:26.233376 5515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:13:26.233554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:26.233431 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:13:27.232614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:27.232580 5515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:13:27.232832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:27.232641 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:13:27.382132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:27.382089 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:28.232717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:28.232679 5515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:13:28.232917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:28.232737 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:13:29.018295 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:13:29.015465 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:13:29.148349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:29.146729 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:29.232531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:29.232492 5515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:13:29.232745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:29.232567 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:13:30.232919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:30.232882 5515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:13:30.233368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:30.232951 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:13:30.918111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:30.918063 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:31.158792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:31.158752 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:31.234862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:31.234759 5515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:13:31.235487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:31.235437 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:13:32.237994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:32.237966 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:13:32.238426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:32.238351 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:13:32.239155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:32.239133 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:13:32.905605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:32.905285 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:33.320342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:33.320209 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:13:34.639128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:34.639085 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:35.932916 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:35.932878 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:36.524389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:36.524349 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:38.144728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:38.144667 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:38.368112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:38.368080 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:39.958806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:39.958766 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:41.514293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:41.514244 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:43.040860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:43.040367 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:44.718289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:44.714596 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:46.474728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:46.474691 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:46.681899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:46.681849 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:47.121616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:47.121211 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:48.309643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:48.309602 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:49.780297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:49.779823 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:51.287404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:51.274687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:52.902696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:52.902663 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:54.586303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:54.579909 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:56.175164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:56.175123 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:57.688433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:57.688382 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:58.566795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.566758 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8"] Oct 30 13:13:58.570641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.570616 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.599017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.598983 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7a8b38\"/\"auth-test-model-7a8b38-kserve-self-signed-certs\"" Oct 30 13:13:58.604324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.600801 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7a8b38\"/\"kube-root-ca.crt\"" Oct 30 13:13:58.604324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.600930 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7a8b38\"/\"default-dockercfg-7dnrc\"" Oct 30 13:13:58.604324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.600943 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7a8b38\"/\"openshift-service-ca.crt\"" Oct 30 13:13:58.604324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.603968 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8"] Oct 30 13:13:58.726397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.726350 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-dshm\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.726906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.726406 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tls-certs\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.726906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.726652 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-home\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.726906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.726720 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-model-cache\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.726906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.726752 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vc5s\" (UniqueName: \"kubernetes.io/projected/a3d10c25-9170-4c3c-9def-7bcfffc06d39-kube-api-access-5vc5s\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.726906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.726858 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tmp-dir\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.801210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.801174 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:58.828987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.828892 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tmp-dir\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.829160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.829096 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-dshm\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.829160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.829141 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tls-certs\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.829337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.829282 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-home\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.829409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.829337 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-model-cache\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.829409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.829361 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc5s\" (UniqueName: \"kubernetes.io/projected/a3d10c25-9170-4c3c-9def-7bcfffc06d39-kube-api-access-5vc5s\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.829528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.829463 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tmp-dir\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.829738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.829679 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-home\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.829880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.829745 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-model-cache\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.831731 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.831703 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-dshm\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.831898 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.831879 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tls-certs\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.858962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.858916 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vc5s\" (UniqueName: \"kubernetes.io/projected/a3d10c25-9170-4c3c-9def-7bcfffc06d39-kube-api-access-5vc5s\") pod \"auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:58.903388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:58.888592 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:13:59.057639 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:13:59.057585 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:13:59.354162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:59.354019 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:13:59.380078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:59.380051 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8"] Oct 30 13:13:59.380923 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:13:59.380893 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3d10c25_9170_4c3c_9def_7bcfffc06d39.slice/crio-b0319c431bd5a106834f0e5b7927efc4569ee9a8461c3296d0e0bba3f4f03d1d WatchSource:0}: Error finding container b0319c431bd5a106834f0e5b7927efc4569ee9a8461c3296d0e0bba3f4f03d1d: Status 404 returned error can't find the container with id b0319c431bd5a106834f0e5b7927efc4569ee9a8461c3296d0e0bba3f4f03d1d Oct 30 13:13:59.643357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:59.617320 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:13:59.751671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:59.751632 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" event={"ID":"a3d10c25-9170-4c3c-9def-7bcfffc06d39","Type":"ContainerStarted","Data":"c96d10066c8579c733353102014af05aa3042a4bab94a68941949bf9419dce41"} Oct 30 13:13:59.752091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:59.751681 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" event={"ID":"a3d10c25-9170-4c3c-9def-7bcfffc06d39","Type":"ContainerStarted","Data":"b0319c431bd5a106834f0e5b7927efc4569ee9a8461c3296d0e0bba3f4f03d1d"} Oct 30 13:13:59.794408 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:59.794337 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" podStartSLOduration=1.7943176140000001 podStartE2EDuration="1.794317614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:59.793529471 +0000 UTC m=+2457.920330381" watchObservedRunningTime="2026-10-30 13:13:59.794317614 +0000 UTC m=+2457.921118518" Oct 30 13:13:59.893544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:13:59.893455 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:14:01.305095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:01.305057 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:02.889448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:02.885728 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:03.380512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:03.380412 5515 scope.go:122] "RemoveContainer" containerID="5fab7c1e6fc1d129d85475fcc61658021a8dc419487052f467ab8d55df4c2e18" Oct 30 13:14:04.522597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:04.522562 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:04.772528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:04.772484 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:06.046107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:06.044925 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:07.527124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:07.525744 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:08.891466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:08.891427 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:14:08.904040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:08.904009 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:14:09.106045 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:09.106009 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:09.837329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:09.837298 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:14:10.681801 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:10.681768 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:11.444707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:11.444666 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:12.396781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:12.396748 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:14.015968 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:14.015926 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:15.289113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:15.289075 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:15.788209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:15.788167 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:17.436962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:17.436928 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:19.075257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:19.075210 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:19.848066 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:19.848030 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:20.633921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:20.633605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:22.172861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:22.172825 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:23.894753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:23.894682 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:25.393396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:25.393356 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:27.018451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:27.018420 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:28.539370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:28.538316 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:28.989874 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:14:28.989836 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:14:30.125454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:30.125416 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:30.917234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:30.917198 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:31.831444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:31.831403 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:32.949658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:32.949622 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:14:33.402685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:33.402631 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:35.214738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:35.214693 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:36.999644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:36.999555 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:38.692594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:38.692531 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:40.197756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:40.197654 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:41.774138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:41.774106 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:43.289134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:43.289089 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:44.914606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:44.914571 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:46.513024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:46.512988 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:48.085030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:48.084992 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:49.623143 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:49.623104 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:50.346962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.346928 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk"] Oct 30 13:14:50.350674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.350644 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.381149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.381119 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk"] Oct 30 13:14:50.388572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.388535 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-98dabb\"/\"kube-root-ca.crt\"" Oct 30 13:14:50.388572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.388564 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-98dabb\"/\"openshift-service-ca.crt\"" Oct 30 13:14:50.388834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.388659 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-98dabb\"/\"default-dockercfg-gcxk7\"" Oct 30 13:14:50.388834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.388710 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-98dabb\"/\"auth-test-model-98dabb-kserve-self-signed-certs\"" Oct 30 13:14:50.457118 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.457084 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:50.507778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.507741 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wmm9\" (UniqueName: \"kubernetes.io/projected/360f2411-be3c-44bb-81ba-f1c83efd59da-kube-api-access-4wmm9\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.508000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.507803 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-model-cache\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.508078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.508017 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-tmp-dir\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.508167 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.508150 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-home\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.508227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.508196 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/360f2411-be3c-44bb-81ba-f1c83efd59da-tls-certs\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.508336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.508233 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-dshm\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.621613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.621505 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-tmp-dir\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.622488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.622460 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-home\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.623025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.623002 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/360f2411-be3c-44bb-81ba-f1c83efd59da-tls-certs\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.623656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.623633 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-dshm\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.624686 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.624659 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmm9\" (UniqueName: \"kubernetes.io/projected/360f2411-be3c-44bb-81ba-f1c83efd59da-kube-api-access-4wmm9\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.624866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.624850 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-model-cache\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.625482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.625460 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-model-cache\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.625661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.622247 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-tmp-dir\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.625661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.622926 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-home\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.660967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.660932 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmm9\" (UniqueName: \"kubernetes.io/projected/360f2411-be3c-44bb-81ba-f1c83efd59da-kube-api-access-4wmm9\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.664028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.663983 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-dshm\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.664905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.664825 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/360f2411-be3c-44bb-81ba-f1c83efd59da-tls-certs\") pod \"auth-test-model-98dabb-kserve-74f4554c57-rwcbk\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:50.670258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:50.670231 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:51.042437 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:14:51.042400 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod360f2411_be3c_44bb_81ba_f1c83efd59da.slice/crio-31a0078b197aa867323d0493b3c5d4744c371a2d72f670944242b6978fef512e WatchSource:0}: Error finding container 31a0078b197aa867323d0493b3c5d4744c371a2d72f670944242b6978fef512e: Status 404 returned error can't find the container with id 31a0078b197aa867323d0493b3c5d4744c371a2d72f670944242b6978fef512e Oct 30 13:14:51.044177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:51.044149 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk"] Oct 30 13:14:51.177005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:51.176968 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" event={"ID":"360f2411-be3c-44bb-81ba-f1c83efd59da","Type":"ContainerStarted","Data":"f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411"} Oct 30 13:14:51.177185 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:51.177017 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" event={"ID":"360f2411-be3c-44bb-81ba-f1c83efd59da","Type":"ContainerStarted","Data":"31a0078b197aa867323d0493b3c5d4744c371a2d72f670944242b6978fef512e"} Oct 30 13:14:51.220623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:51.220562 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" podStartSLOduration=1.220545617 podStartE2EDuration="1.220545617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:51.219223153 +0000 UTC m=+2509.346024055" watchObservedRunningTime="2026-10-30 13:14:51.220545617 +0000 UTC m=+2509.347346537" Oct 30 13:14:51.328894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:51.328778 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:51.670740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:51.670704 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:14:53.036033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:53.035878 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:53.609506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:53.609473 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:54.616290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:54.614901 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:56.320345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:56.320147 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:58.236328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:58.236287 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:58.484909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:58.484871 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:14:58.992262 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:14:58.992228 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:14:59.615296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:14:59.611131 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:00.224039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.223999 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr"] Oct 30 13:15:00.229111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.229083 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:00.263890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.263853 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr"] Oct 30 13:15:00.270624 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.270583 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv"] Oct 30 13:15:00.274751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.274721 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:00.276429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.276404 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh"] Oct 30 13:15:00.282355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.282246 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:00.288522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.288490 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\"" Oct 30 13:15:00.298116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.298080 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv"] Oct 30 13:15:00.306399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.306363 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh"] Oct 30 13:15:00.332226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.332184 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtk2h\" (UniqueName: \"kubernetes.io/projected/5bfef362-1a1a-417f-8d53-6573b67e2130-kube-api-access-mtk2h\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv\" (UID: \"5bfef362-1a1a-417f-8d53-6573b67e2130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:00.332449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.332244 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf79802-820b-4a8a-b5d6-e554632f09b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh\" (UID: \"1cf79802-820b-4a8a-b5d6-e554632f09b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:00.332449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.332396 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd27l\" (UniqueName: \"kubernetes.io/projected/8735b619-6f32-41db-b5e3-9cf19c6e8a21-kube-api-access-xd27l\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr\" (UID: \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:00.332591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.332445 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8735b619-6f32-41db-b5e3-9cf19c6e8a21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr\" (UID: \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:00.332591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.332533 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfef362-1a1a-417f-8d53-6573b67e2130-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv\" (UID: \"5bfef362-1a1a-417f-8d53-6573b67e2130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:00.332707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.332626 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkq5s\" (UniqueName: \"kubernetes.io/projected/1cf79802-820b-4a8a-b5d6-e554632f09b4-kube-api-access-rkq5s\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh\" (UID: \"1cf79802-820b-4a8a-b5d6-e554632f09b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:00.434302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.434246 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfef362-1a1a-417f-8d53-6573b67e2130-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv\" (UID: \"5bfef362-1a1a-417f-8d53-6573b67e2130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:00.434489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.434332 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkq5s\" (UniqueName: \"kubernetes.io/projected/1cf79802-820b-4a8a-b5d6-e554632f09b4-kube-api-access-rkq5s\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh\" (UID: \"1cf79802-820b-4a8a-b5d6-e554632f09b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:00.434534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.434510 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtk2h\" (UniqueName: \"kubernetes.io/projected/5bfef362-1a1a-417f-8d53-6573b67e2130-kube-api-access-mtk2h\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv\" (UID: \"5bfef362-1a1a-417f-8d53-6573b67e2130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:00.434572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.434556 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf79802-820b-4a8a-b5d6-e554632f09b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh\" (UID: \"1cf79802-820b-4a8a-b5d6-e554632f09b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:00.434616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.434605 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd27l\" (UniqueName: \"kubernetes.io/projected/8735b619-6f32-41db-b5e3-9cf19c6e8a21-kube-api-access-xd27l\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr\" (UID: \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:00.434746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.434703 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8735b619-6f32-41db-b5e3-9cf19c6e8a21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr\" (UID: \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:00.434969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.434946 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfef362-1a1a-417f-8d53-6573b67e2130-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv\" (UID: \"5bfef362-1a1a-417f-8d53-6573b67e2130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:00.435478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.435454 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf79802-820b-4a8a-b5d6-e554632f09b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh\" (UID: \"1cf79802-820b-4a8a-b5d6-e554632f09b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:00.435592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.435572 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8735b619-6f32-41db-b5e3-9cf19c6e8a21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr\" (UID: \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:00.462981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.462953 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkq5s\" (UniqueName: \"kubernetes.io/projected/1cf79802-820b-4a8a-b5d6-e554632f09b4-kube-api-access-rkq5s\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh\" (UID: \"1cf79802-820b-4a8a-b5d6-e554632f09b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:00.467976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.467941 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtk2h\" (UniqueName: \"kubernetes.io/projected/5bfef362-1a1a-417f-8d53-6573b67e2130-kube-api-access-mtk2h\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv\" (UID: \"5bfef362-1a1a-417f-8d53-6573b67e2130\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:00.468124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.468059 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd27l\" (UniqueName: \"kubernetes.io/projected/8735b619-6f32-41db-b5e3-9cf19c6e8a21-kube-api-access-xd27l\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr\" (UID: \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:00.546760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.546670 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:00.593790 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.593748 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:00.615445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.614002 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:00.681365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.671976 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:15:00.698312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.695782 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:15:00.855198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.855165 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr"] Oct 30 13:15:00.855508 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:15:00.855481 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8735b619_6f32_41db_b5e3_9cf19c6e8a21.slice/crio-447d69ae0ef00864d36b6ed77612a7db27b1510b4860ee75e51d1b5dc43b5efc WatchSource:0}: Error finding container 447d69ae0ef00864d36b6ed77612a7db27b1510b4860ee75e51d1b5dc43b5efc: Status 404 returned error can't find the container with id 447d69ae0ef00864d36b6ed77612a7db27b1510b4860ee75e51d1b5dc43b5efc Oct 30 13:15:00.915166 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:15:00.915138 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfef362_1a1a_417f_8d53_6573b67e2130.slice/crio-c3ccfcf6f38fe4b1d64a27e2bdae1e534db09b72baf4ba54c7ddbe8be8550897 WatchSource:0}: Error finding container c3ccfcf6f38fe4b1d64a27e2bdae1e534db09b72baf4ba54c7ddbe8be8550897: Status 404 returned error can't find the container with id c3ccfcf6f38fe4b1d64a27e2bdae1e534db09b72baf4ba54c7ddbe8be8550897 Oct 30 13:15:00.916232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.916201 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv"] Oct 30 13:15:00.965757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:00.965719 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh"] Oct 30 13:15:00.965996 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:15:00.965974 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf79802_820b_4a8a_b5d6_e554632f09b4.slice/crio-fd1f1ec4504d2ec30df6e27b919cff1cf32b4a36dc37f581ee3ff72f2f1f8b4b WatchSource:0}: Error finding container fd1f1ec4504d2ec30df6e27b919cff1cf32b4a36dc37f581ee3ff72f2f1f8b4b: Status 404 returned error can't find the container with id fd1f1ec4504d2ec30df6e27b919cff1cf32b4a36dc37f581ee3ff72f2f1f8b4b Oct 30 13:15:01.315884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:01.294605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:01.315884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:01.309667 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" event={"ID":"8735b619-6f32-41db-b5e3-9cf19c6e8a21","Type":"ContainerStarted","Data":"447d69ae0ef00864d36b6ed77612a7db27b1510b4860ee75e51d1b5dc43b5efc"} Oct 30 13:15:01.315884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:01.312308 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" event={"ID":"1cf79802-820b-4a8a-b5d6-e554632f09b4","Type":"ContainerStarted","Data":"fd1f1ec4504d2ec30df6e27b919cff1cf32b4a36dc37f581ee3ff72f2f1f8b4b"} Oct 30 13:15:01.330591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:01.317107 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" event={"ID":"5bfef362-1a1a-417f-8d53-6573b67e2130","Type":"ContainerStarted","Data":"c3ccfcf6f38fe4b1d64a27e2bdae1e534db09b72baf4ba54c7ddbe8be8550897"} Oct 30 13:15:01.350601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:01.350561 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:15:02.324537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.324503 5515 generic.go:356] "Generic (PLEG): container finished" podID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerID="49b6ec58baaae247c56771e1e8149ba3ff75f4c75a24c5887e7911e0521a05a3" exitCode=60 Oct 30 13:15:02.325050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.324587 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" event={"ID":"5bfef362-1a1a-417f-8d53-6573b67e2130","Type":"ContainerDied","Data":"49b6ec58baaae247c56771e1e8149ba3ff75f4c75a24c5887e7911e0521a05a3"} Oct 30 13:15:02.325050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.324903 5515 scope.go:122] "RemoveContainer" containerID="49b6ec58baaae247c56771e1e8149ba3ff75f4c75a24c5887e7911e0521a05a3" Oct 30 13:15:02.326492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.326465 5515 generic.go:356] "Generic (PLEG): container finished" podID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerID="083f2005775ef4fc510d95dcafb32f4151f1edd8adc3728dc5f6f434c4eb8dcf" exitCode=60 Oct 30 13:15:02.326624 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.326541 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" event={"ID":"8735b619-6f32-41db-b5e3-9cf19c6e8a21","Type":"ContainerDied","Data":"083f2005775ef4fc510d95dcafb32f4151f1edd8adc3728dc5f6f434c4eb8dcf"} Oct 30 13:15:02.326807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.326791 5515 scope.go:122] "RemoveContainer" containerID="083f2005775ef4fc510d95dcafb32f4151f1edd8adc3728dc5f6f434c4eb8dcf" Oct 30 13:15:02.328740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.328597 5515 generic.go:356] "Generic (PLEG): container finished" podID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerID="54c6f3edf1a2cec5b974f2226461e22feac3eb49bfc27cd4a131dfbd806fc7cd" exitCode=60 Oct 30 13:15:02.328740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.328662 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" event={"ID":"1cf79802-820b-4a8a-b5d6-e554632f09b4","Type":"ContainerDied","Data":"54c6f3edf1a2cec5b974f2226461e22feac3eb49bfc27cd4a131dfbd806fc7cd"} Oct 30 13:15:02.328908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.328874 5515 scope.go:122] "RemoveContainer" containerID="54c6f3edf1a2cec5b974f2226461e22feac3eb49bfc27cd4a131dfbd806fc7cd" Oct 30 13:15:02.994225 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:02.994148 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:03.337737 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:03.337700 5515 generic.go:356] "Generic (PLEG): container finished" podID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerID="78b9a15d3776f55e868624faf62875e7567489e332ec3527a48b627d8bd9d6f5" exitCode=60 Oct 30 13:15:03.338217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:03.337750 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" event={"ID":"8735b619-6f32-41db-b5e3-9cf19c6e8a21","Type":"ContainerDied","Data":"78b9a15d3776f55e868624faf62875e7567489e332ec3527a48b627d8bd9d6f5"} Oct 30 13:15:03.338217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:03.337812 5515 scope.go:122] "RemoveContainer" containerID="083f2005775ef4fc510d95dcafb32f4151f1edd8adc3728dc5f6f434c4eb8dcf" Oct 30 13:15:03.338217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:03.338136 5515 scope.go:122] "RemoveContainer" containerID="78b9a15d3776f55e868624faf62875e7567489e332ec3527a48b627d8bd9d6f5" Oct 30 13:15:03.338577 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:03.338548 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr_odh-ai-gateway-infra(8735b619-6f32-41db-b5e3-9cf19c6e8a21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" Oct 30 13:15:03.340309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:03.340278 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" event={"ID":"1cf79802-820b-4a8a-b5d6-e554632f09b4","Type":"ContainerStarted","Data":"12e13164deb5cfcf59088150e0c6976e88f42021a8ecf1f4d8bedb2c25787002"} Oct 30 13:15:03.342518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:03.342485 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" event={"ID":"5bfef362-1a1a-417f-8d53-6573b67e2130","Type":"ContainerStarted","Data":"026f353a8d6792c4c490dbb8931214eece2acd62fcf2bc1faf89e465dace3f7d"} Oct 30 13:15:03.431542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:03.431477 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" podStartSLOduration=3.431456371 podStartE2EDuration="3.431456371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.43000313 +0000 UTC m=+2521.556804030" watchObservedRunningTime="2026-10-30 13:15:03.431456371 +0000 UTC m=+2521.558257271" Oct 30 13:15:03.472031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:03.471968 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" podStartSLOduration=3.471954416 podStartE2EDuration="3.471954416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.470457986 +0000 UTC m=+2521.597258883" watchObservedRunningTime="2026-10-30 13:15:03.471954416 +0000 UTC m=+2521.598755314" Oct 30 13:15:04.354712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.354679 5515 generic.go:356] "Generic (PLEG): container finished" podID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerID="12e13164deb5cfcf59088150e0c6976e88f42021a8ecf1f4d8bedb2c25787002" exitCode=60 Oct 30 13:15:04.355198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.354764 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" event={"ID":"1cf79802-820b-4a8a-b5d6-e554632f09b4","Type":"ContainerDied","Data":"12e13164deb5cfcf59088150e0c6976e88f42021a8ecf1f4d8bedb2c25787002"} Oct 30 13:15:04.355198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.354874 5515 scope.go:122] "RemoveContainer" containerID="54c6f3edf1a2cec5b974f2226461e22feac3eb49bfc27cd4a131dfbd806fc7cd" Oct 30 13:15:04.355579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.355557 5515 scope.go:122] "RemoveContainer" containerID="12e13164deb5cfcf59088150e0c6976e88f42021a8ecf1f4d8bedb2c25787002" Oct 30 13:15:04.356045 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:04.356015 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh_odh-ai-gateway-infra(1cf79802-820b-4a8a-b5d6-e554632f09b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" Oct 30 13:15:04.358546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.358524 5515 generic.go:356] "Generic (PLEG): container finished" podID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerID="026f353a8d6792c4c490dbb8931214eece2acd62fcf2bc1faf89e465dace3f7d" exitCode=60 Oct 30 13:15:04.358661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.358605 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" event={"ID":"5bfef362-1a1a-417f-8d53-6573b67e2130","Type":"ContainerDied","Data":"026f353a8d6792c4c490dbb8931214eece2acd62fcf2bc1faf89e465dace3f7d"} Oct 30 13:15:04.358853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.358834 5515 scope.go:122] "RemoveContainer" containerID="026f353a8d6792c4c490dbb8931214eece2acd62fcf2bc1faf89e465dace3f7d" Oct 30 13:15:04.359228 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:04.359203 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv_odh-ai-gateway-infra(5bfef362-1a1a-417f-8d53-6573b67e2130)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" Oct 30 13:15:04.360874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.360858 5515 scope.go:122] "RemoveContainer" containerID="78b9a15d3776f55e868624faf62875e7567489e332ec3527a48b627d8bd9d6f5" Oct 30 13:15:04.361171 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:04.361152 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr_odh-ai-gateway-infra(8735b619-6f32-41db-b5e3-9cf19c6e8a21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" Oct 30 13:15:04.368186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.368165 5515 scope.go:122] "RemoveContainer" containerID="49b6ec58baaae247c56771e1e8149ba3ff75f4c75a24c5887e7911e0521a05a3" Oct 30 13:15:04.684292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:04.678374 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:05.370056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:05.370026 5515 scope.go:122] "RemoveContainer" containerID="12e13164deb5cfcf59088150e0c6976e88f42021a8ecf1f4d8bedb2c25787002" Oct 30 13:15:05.370524 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:05.370379 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh_odh-ai-gateway-infra(1cf79802-820b-4a8a-b5d6-e554632f09b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" Oct 30 13:15:05.371655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:05.371638 5515 scope.go:122] "RemoveContainer" containerID="026f353a8d6792c4c490dbb8931214eece2acd62fcf2bc1faf89e465dace3f7d" Oct 30 13:15:05.371925 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:05.371907 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv_odh-ai-gateway-infra(5bfef362-1a1a-417f-8d53-6573b67e2130)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" Oct 30 13:15:06.527187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:06.527150 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:08.515628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:08.515595 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:10.345347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:10.345312 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:11.326187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:11.326150 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:11.566800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:11.566761 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:12.291308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:12.290019 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:13.976599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:13.968924 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:14.994628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:14.994593 5515 scope.go:122] "RemoveContainer" containerID="026f353a8d6792c4c490dbb8931214eece2acd62fcf2bc1faf89e465dace3f7d" Oct 30 13:15:15.541454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:15.541409 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" event={"ID":"5bfef362-1a1a-417f-8d53-6573b67e2130","Type":"ContainerStarted","Data":"35c0142686c95e880a8ff10ffc76de1cb23a6165705d38a5a04c38557f5ee20e"} Oct 30 13:15:15.667931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:15.667890 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:16.552670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:16.552627 5515 generic.go:356] "Generic (PLEG): container finished" podID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerID="35c0142686c95e880a8ff10ffc76de1cb23a6165705d38a5a04c38557f5ee20e" exitCode=60 Oct 30 13:15:16.553149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:16.552698 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" event={"ID":"5bfef362-1a1a-417f-8d53-6573b67e2130","Type":"ContainerDied","Data":"35c0142686c95e880a8ff10ffc76de1cb23a6165705d38a5a04c38557f5ee20e"} Oct 30 13:15:16.553149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:16.552779 5515 scope.go:122] "RemoveContainer" containerID="026f353a8d6792c4c490dbb8931214eece2acd62fcf2bc1faf89e465dace3f7d" Oct 30 13:15:16.553331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:16.553220 5515 scope.go:122] "RemoveContainer" containerID="35c0142686c95e880a8ff10ffc76de1cb23a6165705d38a5a04c38557f5ee20e" Oct 30 13:15:16.553696 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:16.553668 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv_odh-ai-gateway-infra(5bfef362-1a1a-417f-8d53-6573b67e2130)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" Oct 30 13:15:16.598972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:16.598929 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv"] Oct 30 13:15:16.994432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:16.994393 5515 scope.go:122] "RemoveContainer" containerID="78b9a15d3776f55e868624faf62875e7567489e332ec3527a48b627d8bd9d6f5" Oct 30 13:15:16.994623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:16.994495 5515 scope.go:122] "RemoveContainer" containerID="12e13164deb5cfcf59088150e0c6976e88f42021a8ecf1f4d8bedb2c25787002" Oct 30 13:15:17.562493 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.562397 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" event={"ID":"8735b619-6f32-41db-b5e3-9cf19c6e8a21","Type":"ContainerStarted","Data":"d4d8aa6758a4ee34d298a9b5cdde756159f4ad0748dbcd157f92c96705c2bbde"} Oct 30 13:15:17.564558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.564529 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" event={"ID":"1cf79802-820b-4a8a-b5d6-e554632f09b4","Type":"ContainerStarted","Data":"788a2043a8337c747ebd308d9b1dc12b22f44ace5d352ddbcd2b29c6411506f0"} Oct 30 13:15:17.618293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.616740 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" podStartSLOduration=17.616709713 podStartE2EDuration="17.616709713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:17.604510685 +0000 UTC m=+2535.731311586" watchObservedRunningTime="2026-10-30 13:15:17.616709713 +0000 UTC m=+2535.743510612" Oct 30 13:15:17.658291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.654071 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:17.822791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.822717 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:17.919874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.919840 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfef362-1a1a-417f-8d53-6573b67e2130-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfef362-1a1a-417f-8d53-6573b67e2130-maas-api-service-ca\") pod \"5bfef362-1a1a-417f-8d53-6573b67e2130\" (UID: \"5bfef362-1a1a-417f-8d53-6573b67e2130\") " Oct 30 13:15:17.920092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.920074 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfef362-1a1a-417f-8d53-6573b67e2130-kube-api-access-mtk2h\" (UniqueName: \"kubernetes.io/projected/5bfef362-1a1a-417f-8d53-6573b67e2130-kube-api-access-mtk2h\") pod \"5bfef362-1a1a-417f-8d53-6573b67e2130\" (UID: \"5bfef362-1a1a-417f-8d53-6573b67e2130\") " Oct 30 13:15:17.920246 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.920221 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfef362-1a1a-417f-8d53-6573b67e2130-maas-api-service-ca" pod "5bfef362-1a1a-417f-8d53-6573b67e2130" (UID: "5bfef362-1a1a-417f-8d53-6573b67e2130"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:17.920753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.920732 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfef362-1a1a-417f-8d53-6573b67e2130-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:15:17.994348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:17.993912 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfef362-1a1a-417f-8d53-6573b67e2130-kube-api-access-mtk2h" pod "5bfef362-1a1a-417f-8d53-6573b67e2130" (UID: "5bfef362-1a1a-417f-8d53-6573b67e2130"). InnerVolumeSpecName "kube-api-access-mtk2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:18.022235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.022207 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtk2h\" (UniqueName: \"kubernetes.io/projected/5bfef362-1a1a-417f-8d53-6573b67e2130-kube-api-access-mtk2h\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:15:18.574797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.574761 5515 generic.go:356] "Generic (PLEG): container finished" podID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerID="788a2043a8337c747ebd308d9b1dc12b22f44ace5d352ddbcd2b29c6411506f0" exitCode=60 Oct 30 13:15:18.575338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.574819 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" event={"ID":"1cf79802-820b-4a8a-b5d6-e554632f09b4","Type":"ContainerDied","Data":"788a2043a8337c747ebd308d9b1dc12b22f44ace5d352ddbcd2b29c6411506f0"} Oct 30 13:15:18.575338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.574885 5515 scope.go:122] "RemoveContainer" containerID="12e13164deb5cfcf59088150e0c6976e88f42021a8ecf1f4d8bedb2c25787002" Oct 30 13:15:18.575490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.575355 5515 scope.go:122] "RemoveContainer" containerID="788a2043a8337c747ebd308d9b1dc12b22f44ace5d352ddbcd2b29c6411506f0" Oct 30 13:15:18.575763 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:18.575739 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh_odh-ai-gateway-infra(1cf79802-820b-4a8a-b5d6-e554632f09b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" Oct 30 13:15:18.576604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.576581 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" Oct 30 13:15:18.576726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.576610 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv" event={"ID":"5bfef362-1a1a-417f-8d53-6573b67e2130","Type":"ContainerDied","Data":"c3ccfcf6f38fe4b1d64a27e2bdae1e534db09b72baf4ba54c7ddbe8be8550897"} Oct 30 13:15:18.579026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.579000 5515 generic.go:356] "Generic (PLEG): container finished" podID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerID="d4d8aa6758a4ee34d298a9b5cdde756159f4ad0748dbcd157f92c96705c2bbde" exitCode=60 Oct 30 13:15:18.579182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.579078 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" event={"ID":"8735b619-6f32-41db-b5e3-9cf19c6e8a21","Type":"ContainerDied","Data":"d4d8aa6758a4ee34d298a9b5cdde756159f4ad0748dbcd157f92c96705c2bbde"} Oct 30 13:15:18.579555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.579536 5515 scope.go:122] "RemoveContainer" containerID="d4d8aa6758a4ee34d298a9b5cdde756159f4ad0748dbcd157f92c96705c2bbde" Oct 30 13:15:18.580204 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:18.579883 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr_odh-ai-gateway-infra(8735b619-6f32-41db-b5e3-9cf19c6e8a21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" Oct 30 13:15:18.588379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.588359 5515 scope.go:122] "RemoveContainer" containerID="35c0142686c95e880a8ff10ffc76de1cb23a6165705d38a5a04c38557f5ee20e" Oct 30 13:15:18.598514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.598488 5515 scope.go:122] "RemoveContainer" containerID="78b9a15d3776f55e868624faf62875e7567489e332ec3527a48b627d8bd9d6f5" Oct 30 13:15:18.615047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.615016 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr"] Oct 30 13:15:18.654298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.654231 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv"] Oct 30 13:15:18.660212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.660181 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-517a25-29889435-v47dv"] Oct 30 13:15:18.666430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:18.666403 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh"] Oct 30 13:15:19.742515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.742475 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:19.844970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.844941 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:19.889694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.889627 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:19.952129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.952096 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf79802-820b-4a8a-b5d6-e554632f09b4-kube-api-access-rkq5s\" (UniqueName: \"kubernetes.io/projected/1cf79802-820b-4a8a-b5d6-e554632f09b4-kube-api-access-rkq5s\") pod \"1cf79802-820b-4a8a-b5d6-e554632f09b4\" (UID: \"1cf79802-820b-4a8a-b5d6-e554632f09b4\") " Oct 30 13:15:19.952403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.952146 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8735b619-6f32-41db-b5e3-9cf19c6e8a21-kube-api-access-xd27l\" (UniqueName: \"kubernetes.io/projected/8735b619-6f32-41db-b5e3-9cf19c6e8a21-kube-api-access-xd27l\") pod \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\" (UID: \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\") " Oct 30 13:15:19.952403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.952322 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8735b619-6f32-41db-b5e3-9cf19c6e8a21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8735b619-6f32-41db-b5e3-9cf19c6e8a21-maas-api-service-ca\") pod \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\" (UID: \"8735b619-6f32-41db-b5e3-9cf19c6e8a21\") " Oct 30 13:15:19.952598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.952408 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf79802-820b-4a8a-b5d6-e554632f09b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf79802-820b-4a8a-b5d6-e554632f09b4-maas-api-service-ca\") pod \"1cf79802-820b-4a8a-b5d6-e554632f09b4\" (UID: \"1cf79802-820b-4a8a-b5d6-e554632f09b4\") " Oct 30 13:15:19.952946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.952883 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8735b619-6f32-41db-b5e3-9cf19c6e8a21-maas-api-service-ca" pod "8735b619-6f32-41db-b5e3-9cf19c6e8a21" (UID: "8735b619-6f32-41db-b5e3-9cf19c6e8a21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:19.953161 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.953140 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf79802-820b-4a8a-b5d6-e554632f09b4-maas-api-service-ca" pod "1cf79802-820b-4a8a-b5d6-e554632f09b4" (UID: "1cf79802-820b-4a8a-b5d6-e554632f09b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:19.953325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.953302 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8735b619-6f32-41db-b5e3-9cf19c6e8a21-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:15:19.953392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.953330 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf79802-820b-4a8a-b5d6-e554632f09b4-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:15:19.954765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.954740 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf79802-820b-4a8a-b5d6-e554632f09b4-kube-api-access-rkq5s" pod "1cf79802-820b-4a8a-b5d6-e554632f09b4" (UID: "1cf79802-820b-4a8a-b5d6-e554632f09b4"). InnerVolumeSpecName "kube-api-access-rkq5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:19.955220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:19.955200 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8735b619-6f32-41db-b5e3-9cf19c6e8a21-kube-api-access-xd27l" pod "8735b619-6f32-41db-b5e3-9cf19c6e8a21" (UID: "8735b619-6f32-41db-b5e3-9cf19c6e8a21"). InnerVolumeSpecName "kube-api-access-xd27l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:20.006226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.006183 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" path="/var/lib/kubelet/pods/5bfef362-1a1a-417f-8d53-6573b67e2130/volumes" Oct 30 13:15:20.055049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.055006 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkq5s\" (UniqueName: \"kubernetes.io/projected/1cf79802-820b-4a8a-b5d6-e554632f09b4-kube-api-access-rkq5s\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.055049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.055045 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd27l\" (UniqueName: \"kubernetes.io/projected/8735b619-6f32-41db-b5e3-9cf19c6e8a21-kube-api-access-xd27l\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:15:20.460695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.460655 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:20.652500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.652469 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" Oct 30 13:15:20.652694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.652510 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr" event={"ID":"8735b619-6f32-41db-b5e3-9cf19c6e8a21","Type":"ContainerDied","Data":"447d69ae0ef00864d36b6ed77612a7db27b1510b4860ee75e51d1b5dc43b5efc"} Oct 30 13:15:20.652694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.652609 5515 scope.go:122] "RemoveContainer" containerID="d4d8aa6758a4ee34d298a9b5cdde756159f4ad0748dbcd157f92c96705c2bbde" Oct 30 13:15:20.654197 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.654171 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" Oct 30 13:15:20.654349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.654163 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh" event={"ID":"1cf79802-820b-4a8a-b5d6-e554632f09b4","Type":"ContainerDied","Data":"fd1f1ec4504d2ec30df6e27b919cff1cf32b4a36dc37f581ee3ff72f2f1f8b4b"} Oct 30 13:15:20.663884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.663857 5515 scope.go:122] "RemoveContainer" containerID="788a2043a8337c747ebd308d9b1dc12b22f44ace5d352ddbcd2b29c6411506f0" Oct 30 13:15:20.723678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.723592 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr"] Oct 30 13:15:20.726903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.726879 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-92855c-29889435-nvwcr"] Oct 30 13:15:20.783428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.783398 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh"] Oct 30 13:15:20.791092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.791065 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889435-qnswh"] Oct 30 13:15:20.794360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:20.794329 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:21.605922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:21.605833 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:22.005059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:22.005010 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" path="/var/lib/kubelet/pods/1cf79802-820b-4a8a-b5d6-e554632f09b4/volumes" Oct 30 13:15:22.005586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:22.005565 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" path="/var/lib/kubelet/pods/8735b619-6f32-41db-b5e3-9cf19c6e8a21/volumes" Oct 30 13:15:22.119680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:22.119647 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:22.338015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:22.337877 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:23.396650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:23.394071 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:24.981309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:24.979058 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:26.579949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:26.579915 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:28.383636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:28.383533 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:28.990985 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:28.990949 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:15:30.092325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:30.092288 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:31.673293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:31.673047 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:33.380475 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:33.380436 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:35.009299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:35.005053 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:36.732650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:36.732568 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:37.785253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:37.785207 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:15:38.542075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:38.541577 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:40.279186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:40.279155 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:41.959016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:41.958978 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:43.656568 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:43.656534 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:45.336628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:45.336592 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:46.928318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:46.928227 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:48.195771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:48.195731 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:48.608395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:48.607801 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:50.312732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:50.312695 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:51.898112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:51.898069 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:53.621259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:53.621225 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:55.245698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:55.245667 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:56.933764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:56.933732 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:58.266659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:58.266626 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6"] Oct 30 13:15:58.267158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:58.266828 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" podUID="0c75c816-7912-4460-b898-ad9da761027b" containerName="maas-api" containerID="cri-o://8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e" gracePeriod=30 Oct 30 13:15:58.438228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:58.438194 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:58.991611 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:15:58.991574 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:15:59.017596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.017562 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:59.250556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.250471 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:59.544358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.544334 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:15:59.572701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.572671 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c75c816-7912-4460-b898-ad9da761027b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c75c816-7912-4460-b898-ad9da761027b-openshift-service-ca\") pod \"0c75c816-7912-4460-b898-ad9da761027b\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " Oct 30 13:15:59.572844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.572738 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c75c816-7912-4460-b898-ad9da761027b-kube-api-access-8d5lz\" (UniqueName: \"kubernetes.io/projected/0c75c816-7912-4460-b898-ad9da761027b-kube-api-access-8d5lz\") pod \"0c75c816-7912-4460-b898-ad9da761027b\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " Oct 30 13:15:59.572925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.572868 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-metrics-certs\") pod \"0c75c816-7912-4460-b898-ad9da761027b\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " Oct 30 13:15:59.573005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.572982 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c75c816-7912-4460-b898-ad9da761027b-openshift-service-ca" pod "0c75c816-7912-4460-b898-ad9da761027b" (UID: "0c75c816-7912-4460-b898-ad9da761027b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:59.573402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.573379 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c75c816-7912-4460-b898-ad9da761027b-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:15:59.574869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.574850 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-metrics-certs" pod "0c75c816-7912-4460-b898-ad9da761027b" (UID: "0c75c816-7912-4460-b898-ad9da761027b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:59.574869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.574858 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c75c816-7912-4460-b898-ad9da761027b-kube-api-access-8d5lz" pod "0c75c816-7912-4460-b898-ad9da761027b" (UID: "0c75c816-7912-4460-b898-ad9da761027b"). InnerVolumeSpecName "kube-api-access-8d5lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:59.674619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.674582 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls\") pod \"0c75c816-7912-4460-b898-ad9da761027b\" (UID: \"0c75c816-7912-4460-b898-ad9da761027b\") " Oct 30 13:15:59.675203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.675185 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d5lz\" (UniqueName: \"kubernetes.io/projected/0c75c816-7912-4460-b898-ad9da761027b-kube-api-access-8d5lz\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:15:59.675203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.675202 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:15:59.676721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.676697 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls" pod "0c75c816-7912-4460-b898-ad9da761027b" (UID: "0c75c816-7912-4460-b898-ad9da761027b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:59.773682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.773585 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:15:59.777467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:15:59.777441 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c75c816-7912-4460-b898-ad9da761027b-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:00.040143 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:00.040035 5515 generic.go:356] "Generic (PLEG): container finished" podID="0c75c816-7912-4460-b898-ad9da761027b" containerID="8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e" exitCode=0 Oct 30 13:16:00.040356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:00.040137 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" Oct 30 13:16:00.040356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:00.040165 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" event={"ID":"0c75c816-7912-4460-b898-ad9da761027b","Type":"ContainerDied","Data":"8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e"} Oct 30 13:16:00.040356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:00.040210 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6" event={"ID":"0c75c816-7912-4460-b898-ad9da761027b","Type":"ContainerDied","Data":"5267607be752c93a90253858d83a2ad888e9e748f892734a4a7fa60611b9150c"} Oct 30 13:16:00.040356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:00.040233 5515 scope.go:122] "RemoveContainer" containerID="8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e" Oct 30 13:16:00.056738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:00.056711 5515 scope.go:122] "RemoveContainer" containerID="8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e" Oct 30 13:16:00.057116 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:16:00.057092 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e\": container with ID starting with 8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e not found: ID does not exist" containerID="8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e" Oct 30 13:16:00.057179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:00.057127 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e"} err="failed to get container status \"8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e\": rpc error: code = NotFound desc = could not find container \"8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e\": container with ID starting with 8f8a04a512cc552ee699f4a881f7c8cd59d4309a1da466ae005948f0e464156e not found: ID does not exist" Oct 30 13:16:00.096218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:00.096174 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6"] Oct 30 13:16:00.102433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:00.102401 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2fdf91-668dcdf9cc-cqhq6"] Oct 30 13:16:01.024771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:01.024731 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:02.005110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:02.005076 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c75c816-7912-4460-b898-ad9da761027b" path="/var/lib/kubelet/pods/0c75c816-7912-4460-b898-ad9da761027b/volumes" Oct 30 13:16:02.764209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:02.764180 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:04.540793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:04.540755 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:04.815564 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:04.815468 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:06.295676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:06.295603 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:08.018967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:08.018934 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:09.623384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:09.623225 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:11.314383 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:11.314344 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:11.660690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:11.660649 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:13.078642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:13.078612 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:14.802868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:14.802823 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:16.539181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:16.539137 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:18.141846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:18.141802 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:19.750694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:19.750662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:21.410137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:21.410105 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:23.045933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:23.045901 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:24.735334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:24.735297 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:26.337742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:26.337702 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:27.925452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:27.925397 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:28.990537 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:16:28.990499 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:16:29.513859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:29.513796 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:31.211718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:31.211682 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:31.526507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:31.526427 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:16:33.418164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:33.418122 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:34.987260 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:34.987218 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:36.690874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:36.690822 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:38.357869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:38.357835 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:40.015100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:40.015060 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:41.614530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:41.614498 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:41.841563 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:41.841530 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:42.616296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:42.616189 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:43.425012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:43.424654 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:45.082608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:45.082568 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b"] Oct 30 13:16:45.100713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:45.093601 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eadbb5ef-mr55b"] Oct 30 13:16:45.106003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:45.105971 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:45.907943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:45.907844 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:46.003664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:46.003629 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60244c12-b78e-4179-bc95-f3da973e6421" path="/var/lib/kubelet/pods/60244c12-b78e-4179-bc95-f3da973e6421/volumes" Oct 30 13:16:46.835673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:46.835637 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:48.561799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:48.561765 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:49.064437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:49.064402 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl"] Oct 30 13:16:49.064660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:49.064614 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" containerID="cri-o://7b8b2e929d2926ab8e63128f20ee52d8e2fc8e3b88fd4f3a4cfbb5441c8c99a0" gracePeriod=30 Oct 30 13:16:50.335076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:50.335004 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:52.049599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:52.049558 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:53.842175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:53.842117 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:54.515489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.515439 5515 generic.go:356] "Generic (PLEG): container finished" podID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerID="7b8b2e929d2926ab8e63128f20ee52d8e2fc8e3b88fd4f3a4cfbb5441c8c99a0" exitCode=0 Oct 30 13:16:54.515971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.515931 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" event={"ID":"1ff0adac-2f41-4528-ab90-ff59a93db902","Type":"ContainerDied","Data":"7b8b2e929d2926ab8e63128f20ee52d8e2fc8e3b88fd4f3a4cfbb5441c8c99a0"} Oct 30 13:16:54.516196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.516163 5515 scope.go:122] "RemoveContainer" containerID="b8097d89590850f890f2e2a320c0b126a706f1daf2b5a5428896aab65133e1aa" Oct 30 13:16:54.835285 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.835141 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:16:54.887602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.887564 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-podinfo\") pod \"1ff0adac-2f41-4528-ab90-ff59a93db902\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " Oct 30 13:16:54.888131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.887621 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-token\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-token\") pod \"1ff0adac-2f41-4528-ab90-ff59a93db902\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " Oct 30 13:16:54.888131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.887725 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-certs\") pod \"1ff0adac-2f41-4528-ab90-ff59a93db902\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " Oct 30 13:16:54.888131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.887773 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-credential-socket\") pod \"1ff0adac-2f41-4528-ab90-ff59a93db902\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " Oct 30 13:16:54.888131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.887796 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-data\") pod \"1ff0adac-2f41-4528-ab90-ff59a93db902\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " Oct 30 13:16:54.888131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.887827 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff0adac-2f41-4528-ab90-ff59a93db902-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff0adac-2f41-4528-ab90-ff59a93db902-istiod-ca-cert\") pod \"1ff0adac-2f41-4528-ab90-ff59a93db902\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " Oct 30 13:16:54.888131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.887928 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-envoy\") pod \"1ff0adac-2f41-4528-ab90-ff59a93db902\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " Oct 30 13:16:54.888131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888052 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-socket\") pod \"1ff0adac-2f41-4528-ab90-ff59a93db902\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " Oct 30 13:16:54.888131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888059 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-certs" pod "1ff0adac-2f41-4528-ab90-ff59a93db902" (UID: "1ff0adac-2f41-4528-ab90-ff59a93db902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.888131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888111 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-credential-socket" pod "1ff0adac-2f41-4528-ab90-ff59a93db902" (UID: "1ff0adac-2f41-4528-ab90-ff59a93db902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.888913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888139 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-kube-api-access-55vtj\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-kube-api-access-55vtj\") pod \"1ff0adac-2f41-4528-ab90-ff59a93db902\" (UID: \"1ff0adac-2f41-4528-ab90-ff59a93db902\") " Oct 30 13:16:54.888913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888217 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff0adac-2f41-4528-ab90-ff59a93db902-istiod-ca-cert" pod "1ff0adac-2f41-4528-ab90-ff59a93db902" (UID: "1ff0adac-2f41-4528-ab90-ff59a93db902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:54.888913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888355 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-socket" pod "1ff0adac-2f41-4528-ab90-ff59a93db902" (UID: "1ff0adac-2f41-4528-ab90-ff59a93db902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.888913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888672 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-data" pod "1ff0adac-2f41-4528-ab90-ff59a93db902" (UID: "1ff0adac-2f41-4528-ab90-ff59a93db902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.889141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888961 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.889141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888976 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.889141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888986 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.889141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.888995 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff0adac-2f41-4528-ab90-ff59a93db902-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.889141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.889002 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.890482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.890415 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-token" pod "1ff0adac-2f41-4528-ab90-ff59a93db902" (UID: "1ff0adac-2f41-4528-ab90-ff59a93db902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:54.890482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.890433 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-kube-api-access-55vtj" pod "1ff0adac-2f41-4528-ab90-ff59a93db902" (UID: "1ff0adac-2f41-4528-ab90-ff59a93db902"). InnerVolumeSpecName "kube-api-access-55vtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:54.890666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.890481 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-podinfo" pod "1ff0adac-2f41-4528-ab90-ff59a93db902" (UID: "1ff0adac-2f41-4528-ab90-ff59a93db902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:54.891203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.891183 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-envoy" pod "1ff0adac-2f41-4528-ab90-ff59a93db902" (UID: "1ff0adac-2f41-4528-ab90-ff59a93db902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.990669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.990626 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.990669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.990661 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55vtj\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-kube-api-access-55vtj\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.990669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.990671 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.990669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:54.990679 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff0adac-2f41-4528-ab90-ff59a93db902-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:16:55.525300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:55.525240 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" Oct 30 13:16:55.525300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:55.525283 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl" event={"ID":"1ff0adac-2f41-4528-ab90-ff59a93db902","Type":"ContainerDied","Data":"58d73640014f590727f65ca84b47634018c01511c147e85215cef9c3383e95ff"} Oct 30 13:16:55.525575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:55.525353 5515 scope.go:122] "RemoveContainer" containerID="7b8b2e929d2926ab8e63128f20ee52d8e2fc8e3b88fd4f3a4cfbb5441c8c99a0" Oct 30 13:16:55.595047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:55.595013 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl"] Oct 30 13:16:55.607676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:55.607643 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-2fdf91-openshift-default-575c5986c7-s6jpl"] Oct 30 13:16:55.746670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:55.746619 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:56.002694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:56.002662 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" path="/var/lib/kubelet/pods/1ff0adac-2f41-4528-ab90-ff59a93db902/volumes" Oct 30 13:16:57.560675 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:57.560644 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:16:58.988649 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:16:58.988613 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:16:59.305834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:16:59.305754 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:00.173194 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:00.173155 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m"] Oct 30 13:17:00.438096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:00.437992 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:00.658036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:00.658000 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:01.027848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:01.027810 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:02.758196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:02.758160 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:04.498259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:04.498218 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:05.174414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.174372 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" podUID="87b2d2b8-78d4-4b35-889d-02ebe0c97fd2" containerName="payload-pre-processing" containerID="cri-o://d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4" gracePeriod=25 Oct 30 13:17:05.445651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.445590 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:17:05.497432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.497396 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-kube-api-access-2vtdm\" (UniqueName: \"kubernetes.io/projected/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-kube-api-access-2vtdm\") pod \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\" (UID: \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\") " Oct 30 13:17:05.497590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.497527 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-plugins-config-volume\") pod \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\" (UID: \"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2\") " Oct 30 13:17:05.497953 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.497921 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-plugins-config-volume" pod "87b2d2b8-78d4-4b35-889d-02ebe0c97fd2" (UID: "87b2d2b8-78d4-4b35-889d-02ebe0c97fd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:05.498123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.498100 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:17:05.499551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.499527 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-kube-api-access-2vtdm" pod "87b2d2b8-78d4-4b35-889d-02ebe0c97fd2" (UID: "87b2d2b8-78d4-4b35-889d-02ebe0c97fd2"). InnerVolumeSpecName "kube-api-access-2vtdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:05.599112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.599076 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vtdm\" (UniqueName: \"kubernetes.io/projected/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2-kube-api-access-2vtdm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:17:05.610108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.610079 5515 generic.go:356] "Generic (PLEG): container finished" podID="87b2d2b8-78d4-4b35-889d-02ebe0c97fd2" containerID="d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4" exitCode=0 Oct 30 13:17:05.610282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.610122 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" Oct 30 13:17:05.610282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.610136 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" event={"ID":"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2","Type":"ContainerDied","Data":"d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4"} Oct 30 13:17:05.610282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.610180 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m" event={"ID":"87b2d2b8-78d4-4b35-889d-02ebe0c97fd2","Type":"ContainerDied","Data":"11c534fd68e39668dcdd22fe06bfdec19043a2b37e3e7b9c7b6f3aca2ae38e71"} Oct 30 13:17:05.610282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.610203 5515 scope.go:122] "RemoveContainer" containerID="d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4" Oct 30 13:17:05.620291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.620255 5515 scope.go:122] "RemoveContainer" containerID="d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4" Oct 30 13:17:05.620564 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:17:05.620545 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4\": container with ID starting with d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4 not found: ID does not exist" containerID="d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4" Oct 30 13:17:05.620632 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.620571 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4"} err="failed to get container status \"d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4\": rpc error: code = NotFound desc = could not find container \"d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4\": container with ID starting with d79ccc69faed934569552af3c3b89b1d89eb38ccae65e7632f4bdf1da8684fc4 not found: ID does not exist" Oct 30 13:17:05.669454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.669426 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m"] Oct 30 13:17:05.672571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:05.672549 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca097a-6fbcf4f95-ldc2m"] Oct 30 13:17:06.002526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:06.002494 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b2d2b8-78d4-4b35-889d-02ebe0c97fd2" path="/var/lib/kubelet/pods/87b2d2b8-78d4-4b35-889d-02ebe0c97fd2/volumes" Oct 30 13:17:06.347079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:06.346992 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:08.120811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:08.120718 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:09.840043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:09.840010 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:11.618406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:11.618364 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:11.838569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:11.838535 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:12.777895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:12.777860 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:13.010011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:13.009974 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:13.238610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:13.238579 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:14.845456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:14.845412 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:16.604097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:16.604051 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:18.328972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:18.328932 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:20.026080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:20.026044 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:21.909701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:21.909668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:22.958347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:22.958313 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:17:23.109717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.109678 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph"] Oct 30 13:17:23.109969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.109938 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" podUID="18aaf89b-5970-46e1-b575-bc01f7b96862" containerName="maas-api" containerID="cri-o://4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8" gracePeriod=30 Oct 30 13:17:23.295110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.295071 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:23.611507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.611471 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:17:23.693963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.693859 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls\") pod \"18aaf89b-5970-46e1-b575-bc01f7b96862\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " Oct 30 13:17:23.693963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.693951 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-metrics-certs\") pod \"18aaf89b-5970-46e1-b575-bc01f7b96862\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " Oct 30 13:17:23.694221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.694074 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18aaf89b-5970-46e1-b575-bc01f7b96862-kube-api-access-tf8tb\" (UniqueName: \"kubernetes.io/projected/18aaf89b-5970-46e1-b575-bc01f7b96862-kube-api-access-tf8tb\") pod \"18aaf89b-5970-46e1-b575-bc01f7b96862\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " Oct 30 13:17:23.694301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.694232 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18aaf89b-5970-46e1-b575-bc01f7b96862-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aaf89b-5970-46e1-b575-bc01f7b96862-openshift-service-ca\") pod \"18aaf89b-5970-46e1-b575-bc01f7b96862\" (UID: \"18aaf89b-5970-46e1-b575-bc01f7b96862\") " Oct 30 13:17:23.703309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.703035 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18aaf89b-5970-46e1-b575-bc01f7b96862-openshift-service-ca" pod "18aaf89b-5970-46e1-b575-bc01f7b96862" (UID: "18aaf89b-5970-46e1-b575-bc01f7b96862"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:23.707369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.707016 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls" pod "18aaf89b-5970-46e1-b575-bc01f7b96862" (UID: "18aaf89b-5970-46e1-b575-bc01f7b96862"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:23.710215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.710187 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18aaf89b-5970-46e1-b575-bc01f7b96862-kube-api-access-tf8tb" pod "18aaf89b-5970-46e1-b575-bc01f7b96862" (UID: "18aaf89b-5970-46e1-b575-bc01f7b96862"). InnerVolumeSpecName "kube-api-access-tf8tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:23.722591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.722541 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-metrics-certs" pod "18aaf89b-5970-46e1-b575-bc01f7b96862" (UID: "18aaf89b-5970-46e1-b575-bc01f7b96862"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:17:23.736672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.736591 5515 generic.go:356] "Generic (PLEG): container finished" podID="18aaf89b-5970-46e1-b575-bc01f7b96862" containerID="4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8" exitCode=0 Oct 30 13:17:23.736672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.736641 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" Oct 30 13:17:23.736876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.736668 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" event={"ID":"18aaf89b-5970-46e1-b575-bc01f7b96862","Type":"ContainerDied","Data":"4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8"} Oct 30 13:17:23.736876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.736724 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph" event={"ID":"18aaf89b-5970-46e1-b575-bc01f7b96862","Type":"ContainerDied","Data":"e38a9aa375d84810c14591a5d6440524abcc9e527da4233ab67db8bcc2e4baa1"} Oct 30 13:17:23.736876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.736746 5515 scope.go:122] "RemoveContainer" containerID="4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8" Oct 30 13:17:23.755035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.754464 5515 scope.go:122] "RemoveContainer" containerID="4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8" Oct 30 13:17:23.755035 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:17:23.754813 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8\": container with ID starting with 4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8 not found: ID does not exist" containerID="4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8" Oct 30 13:17:23.755035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.754838 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8"} err="failed to get container status \"4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8\": rpc error: code = NotFound desc = could not find container \"4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8\": container with ID starting with 4eed98eff096d5bcd944210ca8bcad7c7d547adf5bfdffcd31b7b703eff7c2b8 not found: ID does not exist" Oct 30 13:17:23.796488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.796460 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:17:23.796488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.796484 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18aaf89b-5970-46e1-b575-bc01f7b96862-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:17:23.796488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.796494 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8tb\" (UniqueName: \"kubernetes.io/projected/18aaf89b-5970-46e1-b575-bc01f7b96862-kube-api-access-tf8tb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:17:23.796732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.796505 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aaf89b-5970-46e1-b575-bc01f7b96862-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:17:23.797051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.797028 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph"] Oct 30 13:17:23.802694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.802670 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ca097a-765fbcc6d5-hwtph"] Oct 30 13:17:23.893445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:23.893416 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:24.014072 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:24.013981 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18aaf89b-5970-46e1-b575-bc01f7b96862" path="/var/lib/kubelet/pods/18aaf89b-5970-46e1-b575-bc01f7b96862/volumes" Oct 30 13:17:24.252689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:24.252656 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:24.625610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:24.625567 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:26.110257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:26.110219 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:27.844127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:27.844079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:28.988730 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:17:28.988692 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:17:29.526693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:29.526658 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:30.211126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:30.211089 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:30.431978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:30.431948 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:31.103378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:31.103339 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:31.323974 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:31.323931 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:31.643763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:31.643729 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:32.940790 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:32.940757 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:34.719233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:34.719155 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:36.521591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:36.521556 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:38.153561 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:38.153520 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:38.388117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:38.388085 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:38.695887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:38.695847 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:38.946253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:38.946168 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:40.132764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:40.132732 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:41.985701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:41.985429 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:42.231236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:42.231197 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:43.825559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:43.825517 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:44.112841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:44.112754 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:45.644847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:45.644813 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:47.349154 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:47.349121 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:49.244503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:49.244472 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:50.939162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:50.939115 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:51.533576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:51.533509 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:52.724026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:52.723988 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:54.423440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:54.423406 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:55.085350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:55.085303 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj"] Oct 30 13:17:55.085587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:55.085559 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" containerID="cri-o://cf3b0d9f8eb58d5b26abffe6ca3b15d0bd9bd8518499b8d9b176de43170d1eaa" gracePeriod=30 Oct 30 13:17:55.097889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:55.097837 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d"] Oct 30 13:17:55.101315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:55.101284 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r"] Oct 30 13:17:55.101526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:55.101500 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" containerID="cri-o://90d1803e29a61fbe48b6878d4849ee9827abf50df2684aaab1894f099b002105" gracePeriod=30 Oct 30 13:17:55.114789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:55.113164 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p"] Oct 30 13:17:55.114789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:55.113520 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" containerID="cri-o://ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d" gracePeriod=30 Oct 30 13:17:55.142919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:55.142851 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" containerID="cri-o://ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893" gracePeriod=30 Oct 30 13:17:56.305244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:56.305156 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:57.858037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:57.857999 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:58.990627 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:17:58.990587 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:17:59.505014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:59.504963 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:17:59.776760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:59.776661 5515 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:17:59.776760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:17:59.776734 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:18:00.225393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.225360 5515 generic.go:356] "Generic (PLEG): container finished" podID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerID="90d1803e29a61fbe48b6878d4849ee9827abf50df2684aaab1894f099b002105" exitCode=0 Oct 30 13:18:00.225818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.225433 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" event={"ID":"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8","Type":"ContainerDied","Data":"90d1803e29a61fbe48b6878d4849ee9827abf50df2684aaab1894f099b002105"} Oct 30 13:18:00.227099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.227077 5515 generic.go:356] "Generic (PLEG): container finished" podID="9d1018d8-6578-406e-a683-3c183fb963d0" containerID="cf3b0d9f8eb58d5b26abffe6ca3b15d0bd9bd8518499b8d9b176de43170d1eaa" exitCode=0 Oct 30 13:18:00.227223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.227153 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" event={"ID":"9d1018d8-6578-406e-a683-3c183fb963d0","Type":"ContainerDied","Data":"cf3b0d9f8eb58d5b26abffe6ca3b15d0bd9bd8518499b8d9b176de43170d1eaa"} Oct 30 13:18:00.458566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.458390 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:18:00.469431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469397 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-podinfo\") pod \"4c93a624-1411-42ef-a6b6-5a3dfba23243\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " Oct 30 13:18:00.469614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469449 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-kube-api-access-2pcr2\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-kube-api-access-2pcr2\") pod \"4c93a624-1411-42ef-a6b6-5a3dfba23243\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " Oct 30 13:18:00.469614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469488 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-certs\") pod \"4c93a624-1411-42ef-a6b6-5a3dfba23243\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " Oct 30 13:18:00.469614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469519 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-data\") pod \"4c93a624-1411-42ef-a6b6-5a3dfba23243\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " Oct 30 13:18:00.469614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469571 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-envoy\") pod \"4c93a624-1411-42ef-a6b6-5a3dfba23243\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " Oct 30 13:18:00.469857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469618 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-token\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-token\") pod \"4c93a624-1411-42ef-a6b6-5a3dfba23243\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " Oct 30 13:18:00.469857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469691 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-socket\") pod \"4c93a624-1411-42ef-a6b6-5a3dfba23243\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " Oct 30 13:18:00.469857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469739 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c93a624-1411-42ef-a6b6-5a3dfba23243-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c93a624-1411-42ef-a6b6-5a3dfba23243-istiod-ca-cert\") pod \"4c93a624-1411-42ef-a6b6-5a3dfba23243\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " Oct 30 13:18:00.470024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469872 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-credential-socket\") pod \"4c93a624-1411-42ef-a6b6-5a3dfba23243\" (UID: \"4c93a624-1411-42ef-a6b6-5a3dfba23243\") " Oct 30 13:18:00.470024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.469985 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-certs" pod "4c93a624-1411-42ef-a6b6-5a3dfba23243" (UID: "4c93a624-1411-42ef-a6b6-5a3dfba23243"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.470256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.470232 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-data" pod "4c93a624-1411-42ef-a6b6-5a3dfba23243" (UID: "4c93a624-1411-42ef-a6b6-5a3dfba23243"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.470668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.470641 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.470774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.470669 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.470839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.470817 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c93a624-1411-42ef-a6b6-5a3dfba23243-istiod-ca-cert" pod "4c93a624-1411-42ef-a6b6-5a3dfba23243" (UID: "4c93a624-1411-42ef-a6b6-5a3dfba23243"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:00.470912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.470874 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-credential-socket" pod "4c93a624-1411-42ef-a6b6-5a3dfba23243" (UID: "4c93a624-1411-42ef-a6b6-5a3dfba23243"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.471072 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.471041 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-socket" pod "4c93a624-1411-42ef-a6b6-5a3dfba23243" (UID: "4c93a624-1411-42ef-a6b6-5a3dfba23243"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.474737 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.474692 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-kube-api-access-2pcr2" pod "4c93a624-1411-42ef-a6b6-5a3dfba23243" (UID: "4c93a624-1411-42ef-a6b6-5a3dfba23243"). InnerVolumeSpecName "kube-api-access-2pcr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.475344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.475320 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-token" pod "4c93a624-1411-42ef-a6b6-5a3dfba23243" (UID: "4c93a624-1411-42ef-a6b6-5a3dfba23243"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.476445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.476414 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-envoy" pod "4c93a624-1411-42ef-a6b6-5a3dfba23243" (UID: "4c93a624-1411-42ef-a6b6-5a3dfba23243"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.482520 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.481462 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-podinfo" pod "4c93a624-1411-42ef-a6b6-5a3dfba23243" (UID: "4c93a624-1411-42ef-a6b6-5a3dfba23243"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:00.486708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.485109 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 13:18:00.553712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.553683 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:18:00.560552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.560514 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:18:00.571338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571290 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-credential-socket\") pod \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " Oct 30 13:18:00.571338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571342 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-socket\") pod \"9d1018d8-6578-406e-a683-3c183fb963d0\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " Oct 30 13:18:00.571611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571379 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-socket\") pod \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " Oct 30 13:18:00.571611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571432 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-credential-socket\") pod \"9d1018d8-6578-406e-a683-3c183fb963d0\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " Oct 30 13:18:00.571611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571468 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-envoy\") pod \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " Oct 30 13:18:00.571611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571496 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-kube-api-access-j4785\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-kube-api-access-j4785\") pod \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " Oct 30 13:18:00.571611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571533 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-data\") pod \"9d1018d8-6578-406e-a683-3c183fb963d0\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " Oct 30 13:18:00.571611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571587 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-certs\") pod \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571639 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-certs\") pod \"9d1018d8-6578-406e-a683-3c183fb963d0\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571658 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-credential-socket" pod "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" (UID: "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571672 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-socket" pod "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" (UID: "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571680 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-envoy\") pod \"9d1018d8-6578-406e-a683-3c183fb963d0\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571698 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-socket" pod "9d1018d8-6578-406e-a683-3c183fb963d0" (UID: "9d1018d8-6578-406e-a683-3c183fb963d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571747 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-podinfo\") pod \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571841 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-istio-token\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-istio-token\") pod \"9d1018d8-6578-406e-a683-3c183fb963d0\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571877 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d1018d8-6578-406e-a683-3c183fb963d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1018d8-6578-406e-a683-3c183fb963d0-istio-podinfo\") pod \"9d1018d8-6578-406e-a683-3c183fb963d0\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571910 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-credential-socket\") pod \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571843 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-credential-socket" pod "9d1018d8-6578-406e-a683-3c183fb963d0" (UID: "9d1018d8-6578-406e-a683-3c183fb963d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.571943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571950 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-socket\") pod \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571987 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1018d8-6578-406e-a683-3c183fb963d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1018d8-6578-406e-a683-3c183fb963d0-istiod-ca-cert\") pod \"9d1018d8-6578-406e-a683-3c183fb963d0\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572031 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-kube-api-access-v9vbc\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-kube-api-access-v9vbc\") pod \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572091 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-data\") pod \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572144 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-token\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-token\") pod \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572184 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-token\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-token\") pod \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572216 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-data\") pod \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572308 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-kube-api-access-kfjrr\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-kube-api-access-kfjrr\") pod \"9d1018d8-6578-406e-a683-3c183fb963d0\" (UID: \"9d1018d8-6578-406e-a683-3c183fb963d0\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572354 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-podinfo\") pod \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572424 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-certs\") pod \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572451 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istiod-ca-cert\") pod \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " Oct 30 13:18:00.572528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572483 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istiod-ca-cert\") pod \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\" (UID: \"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8\") " Oct 30 13:18:00.573142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572552 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-envoy\") pod \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\" (UID: \"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2\") " Oct 30 13:18:00.573142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571947 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-certs" pod "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" (UID: "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.573142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571932 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-data" pod "9d1018d8-6578-406e-a683-3c183fb963d0" (UID: "9d1018d8-6578-406e-a683-3c183fb963d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.573142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.571969 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-certs" pod "9d1018d8-6578-406e-a683-3c183fb963d0" (UID: "9d1018d8-6578-406e-a683-3c183fb963d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.573142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572166 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-credential-socket" pod "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" (UID: "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.573142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572485 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1018d8-6578-406e-a683-3c183fb963d0-istiod-ca-cert" pod "9d1018d8-6578-406e-a683-3c183fb963d0" (UID: "9d1018d8-6578-406e-a683-3c183fb963d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:00.573142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572723 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-socket" pod "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" (UID: "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.573142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.572956 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-data" pod "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" (UID: "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573416 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573440 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573456 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573471 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573486 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pcr2\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-kube-api-access-2pcr2\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573501 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573552 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573566 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c93a624-1411-42ef-a6b6-5a3dfba23243-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573580 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573594 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573607 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573620 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c93a624-1411-42ef-a6b6-5a3dfba23243-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573631 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573645 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573659 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573675 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1018d8-6578-406e-a683-3c183fb963d0-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573689 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c93a624-1411-42ef-a6b6-5a3dfba23243-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573703 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.573701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573576 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-certs" pod "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" (UID: "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.574750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.573669 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-data" pod "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" (UID: "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.574750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.574534 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istiod-ca-cert" pod "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" (UID: "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:00.574978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.574944 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istiod-ca-cert" pod "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" (UID: "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:00.576125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576096 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-envoy" pod "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" (UID: "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.576466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576441 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-kube-api-access-kfjrr" pod "9d1018d8-6578-406e-a683-3c183fb963d0" (UID: "9d1018d8-6578-406e-a683-3c183fb963d0"). InnerVolumeSpecName "kube-api-access-kfjrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.576574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576469 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-kube-api-access-v9vbc" pod "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" (UID: "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2"). InnerVolumeSpecName "kube-api-access-v9vbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.576574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576515 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-envoy" pod "9d1018d8-6578-406e-a683-3c183fb963d0" (UID: "9d1018d8-6578-406e-a683-3c183fb963d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.576574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576538 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-envoy" pod "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" (UID: "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.576574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576564 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-kube-api-access-j4785" pod "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" (UID: "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8"). InnerVolumeSpecName "kube-api-access-j4785". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.576574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576569 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d1018d8-6578-406e-a683-3c183fb963d0-istio-podinfo" pod "9d1018d8-6578-406e-a683-3c183fb963d0" (UID: "9d1018d8-6578-406e-a683-3c183fb963d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:00.576903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576576 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-podinfo" pod "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" (UID: "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:00.576903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576876 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-token" pod "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" (UID: "ab051748-23bd-4b4d-8b7a-dab99c1b8ec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.577054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.576934 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-podinfo" pod "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" (UID: "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:00.577967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.577942 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-istio-token" pod "9d1018d8-6578-406e-a683-3c183fb963d0" (UID: "9d1018d8-6578-406e-a683-3c183fb963d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.578288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.578236 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-token" pod "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" (UID: "347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.675875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675835 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.675875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675875 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4785\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-kube-api-access-j4785\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675891 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1018d8-6578-406e-a683-3c183fb963d0-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675904 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675916 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675928 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1018d8-6578-406e-a683-3c183fb963d0-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675942 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9vbc\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-kube-api-access-v9vbc\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675957 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675969 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675981 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.675994 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfjrr\" (UniqueName: \"kubernetes.io/projected/9d1018d8-6578-406e-a683-3c183fb963d0-kube-api-access-kfjrr\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.676007 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.676022 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.676035 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.676048 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.676192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:00.676060 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:01.035943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.035810 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:01.237864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.237827 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" event={"ID":"9d1018d8-6578-406e-a683-3c183fb963d0","Type":"ContainerDied","Data":"f5aa2604df59bd368859787513fa991787171634e946a1f59bfd1a938294ae0f"} Oct 30 13:18:01.237864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.237856 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj" Oct 30 13:18:01.238605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.237938 5515 scope.go:122] "RemoveContainer" containerID="cf3b0d9f8eb58d5b26abffe6ca3b15d0bd9bd8518499b8d9b176de43170d1eaa" Oct 30 13:18:01.240873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.240792 5515 generic.go:356] "Generic (PLEG): container finished" podID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerID="ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893" exitCode=0 Oct 30 13:18:01.240873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.240858 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" Oct 30 13:18:01.241048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.241005 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" event={"ID":"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2","Type":"ContainerDied","Data":"ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893"} Oct 30 13:18:01.241117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.241087 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d" event={"ID":"ab051748-23bd-4b4d-8b7a-dab99c1b8ec2","Type":"ContainerDied","Data":"9ad9b4f7d5c1df1dce91722d34b7dc16734eb3502d1b7feaabc2dce9912540a6"} Oct 30 13:18:01.243416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.243391 5515 generic.go:356] "Generic (PLEG): container finished" podID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerID="ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d" exitCode=0 Oct 30 13:18:01.243542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.243443 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" Oct 30 13:18:01.243542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.243481 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" event={"ID":"4c93a624-1411-42ef-a6b6-5a3dfba23243","Type":"ContainerDied","Data":"ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d"} Oct 30 13:18:01.243542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.243529 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p" event={"ID":"4c93a624-1411-42ef-a6b6-5a3dfba23243","Type":"ContainerDied","Data":"fcc31ef5ffc01ea91dde36adeaad2ed39a6cd3ef60c5e083fdff4c4e2e744c42"} Oct 30 13:18:01.246251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.246227 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" event={"ID":"347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8","Type":"ContainerDied","Data":"90c23b4c5ba98826ef80696e4fff596f79855c42a3d936a74befa71b071bca1f"} Oct 30 13:18:01.246375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.246260 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r" Oct 30 13:18:01.253548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.253402 5515 scope.go:122] "RemoveContainer" containerID="ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893" Oct 30 13:18:01.264943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.264924 5515 scope.go:122] "RemoveContainer" containerID="ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893" Oct 30 13:18:01.265244 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:01.265225 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893\": container with ID starting with ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893 not found: ID does not exist" containerID="ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893" Oct 30 13:18:01.265317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.265254 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893"} err="failed to get container status \"ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893\": rpc error: code = NotFound desc = could not find container \"ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893\": container with ID starting with ab8bcc72c30ecd3bdc3b56eec08933e8ca5cc7ddc038a1b2504859e6c1ffc893 not found: ID does not exist" Oct 30 13:18:01.265317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.265286 5515 scope.go:122] "RemoveContainer" containerID="ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d" Oct 30 13:18:01.275499 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.275476 5515 scope.go:122] "RemoveContainer" containerID="ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d" Oct 30 13:18:01.275803 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:01.275786 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d\": container with ID starting with ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d not found: ID does not exist" containerID="ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d" Oct 30 13:18:01.275856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.275816 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d"} err="failed to get container status \"ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d\": rpc error: code = NotFound desc = could not find container \"ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d\": container with ID starting with ac3f97ffc4fcbeba8a56aadcb34391c688140af4996ebc891f4796231d9d1b4d not found: ID does not exist" Oct 30 13:18:01.275856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.275833 5515 scope.go:122] "RemoveContainer" containerID="90d1803e29a61fbe48b6878d4849ee9827abf50df2684aaab1894f099b002105" Oct 30 13:18:01.346192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.341890 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj"] Oct 30 13:18:01.360797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.359921 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:01.369353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.366733 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-glqmj"] Oct 30 13:18:01.441100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.441061 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r"] Oct 30 13:18:01.445786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.445749 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xrq4r"] Oct 30 13:18:01.482615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.482570 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p"] Oct 30 13:18:01.497424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.497388 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-8mg6p"] Oct 30 13:18:01.527403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.527368 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d"] Oct 30 13:18:01.532376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.532339 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-xcz4d"] Oct 30 13:18:01.679844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:01.679809 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:02.008104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:02.008000 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" path="/var/lib/kubelet/pods/347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8/volumes" Oct 30 13:18:02.008659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:02.008637 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" path="/var/lib/kubelet/pods/4c93a624-1411-42ef-a6b6-5a3dfba23243/volumes" Oct 30 13:18:02.009141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:02.009124 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" path="/var/lib/kubelet/pods/9d1018d8-6578-406e-a683-3c183fb963d0/volumes" Oct 30 13:18:02.009966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:02.009944 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" path="/var/lib/kubelet/pods/ab051748-23bd-4b4d-8b7a-dab99c1b8ec2/volumes" Oct 30 13:18:02.228906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:02.228820 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:02.472121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:02.472081 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:03.211012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:03.210980 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:03.470354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:03.470235 5515 scope.go:122] "RemoveContainer" containerID="569b5b699c1bbb9cdad0871fac0cd22b70c7919065400c46b081c18a0082c525" Oct 30 13:18:04.155547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:04.155508 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:04.391972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:04.391933 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:04.628870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:04.628789 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:05.177476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:05.177424 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:06.807205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:06.807116 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:07.102019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:07.101930 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:07.332016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:07.331977 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:08.721726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:08.721686 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:10.509708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:10.509678 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:11.348069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:11.348033 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql"] Oct 30 13:18:11.348291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:11.348246 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" containerID="cri-o://c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a" gracePeriod=30 Oct 30 13:18:11.947000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:11.946962 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:12.407857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:12.407822 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:14.442564 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:14.442530 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:14.607069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:14.607016 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5"] Oct 30 13:18:14.607338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:14.607297 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" containerID="cri-o://6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf" gracePeriod=30 Oct 30 13:18:15.703664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:15.703558 5515 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:18:15.703664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:15.703634 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:18:15.902583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:15.902545 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:16.193749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.193710 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:18:16.634364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.634336 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 13:18:16.736124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.716005 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-certs\") pod \"e5879660-fa09-42c2-ba35-6ef6a6c67930\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " Oct 30 13:18:16.736124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.716057 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-kube-api-access-k2vgn\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-kube-api-access-k2vgn\") pod \"e5879660-fa09-42c2-ba35-6ef6a6c67930\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " Oct 30 13:18:16.736124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.716176 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5879660-fa09-42c2-ba35-6ef6a6c67930-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5879660-fa09-42c2-ba35-6ef6a6c67930-istiod-ca-cert\") pod \"e5879660-fa09-42c2-ba35-6ef6a6c67930\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " Oct 30 13:18:16.736124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.716212 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-credential-socket\") pod \"e5879660-fa09-42c2-ba35-6ef6a6c67930\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " Oct 30 13:18:16.755468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.739720 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5879660-fa09-42c2-ba35-6ef6a6c67930-istiod-ca-cert" pod "e5879660-fa09-42c2-ba35-6ef6a6c67930" (UID: "e5879660-fa09-42c2-ba35-6ef6a6c67930"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:16.755468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.740321 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-credential-socket" pod "e5879660-fa09-42c2-ba35-6ef6a6c67930" (UID: "e5879660-fa09-42c2-ba35-6ef6a6c67930"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:16.755468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.750900 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-kube-api-access-k2vgn" pod "e5879660-fa09-42c2-ba35-6ef6a6c67930" (UID: "e5879660-fa09-42c2-ba35-6ef6a6c67930"). InnerVolumeSpecName "kube-api-access-k2vgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:16.755468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.755370 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-certs" pod "e5879660-fa09-42c2-ba35-6ef6a6c67930" (UID: "e5879660-fa09-42c2-ba35-6ef6a6c67930"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:16.756124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.756082 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-socket" pod "e5879660-fa09-42c2-ba35-6ef6a6c67930" (UID: "e5879660-fa09-42c2-ba35-6ef6a6c67930"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:16.757315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.716262 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-socket\") pod \"e5879660-fa09-42c2-ba35-6ef6a6c67930\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " Oct 30 13:18:16.757315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.756335 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-podinfo\") pod \"e5879660-fa09-42c2-ba35-6ef6a6c67930\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " Oct 30 13:18:16.757315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.756451 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-token\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-token\") pod \"e5879660-fa09-42c2-ba35-6ef6a6c67930\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " Oct 30 13:18:16.757315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.756579 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-data\") pod \"e5879660-fa09-42c2-ba35-6ef6a6c67930\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " Oct 30 13:18:16.757315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.756640 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-envoy\") pod \"e5879660-fa09-42c2-ba35-6ef6a6c67930\" (UID: \"e5879660-fa09-42c2-ba35-6ef6a6c67930\") " Oct 30 13:18:16.758478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.758170 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-data" pod "e5879660-fa09-42c2-ba35-6ef6a6c67930" (UID: "e5879660-fa09-42c2-ba35-6ef6a6c67930"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:16.758919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.758596 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:16.758919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.758627 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vgn\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-kube-api-access-k2vgn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:16.758919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.758641 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5879660-fa09-42c2-ba35-6ef6a6c67930-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:16.758919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.758656 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:16.758919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.758667 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:16.805509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.805450 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-token" pod "e5879660-fa09-42c2-ba35-6ef6a6c67930" (UID: "e5879660-fa09-42c2-ba35-6ef6a6c67930"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:16.810372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.807658 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-podinfo" pod "e5879660-fa09-42c2-ba35-6ef6a6c67930" (UID: "e5879660-fa09-42c2-ba35-6ef6a6c67930"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:16.810576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.810503 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-envoy" pod "e5879660-fa09-42c2-ba35-6ef6a6c67930" (UID: "e5879660-fa09-42c2-ba35-6ef6a6c67930"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:16.860299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.860250 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:16.860299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.860295 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:16.860299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.860304 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:16.860654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:16.860312 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5879660-fa09-42c2-ba35-6ef6a6c67930-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:17.456996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.456964 5515 generic.go:356] "Generic (PLEG): container finished" podID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerID="c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a" exitCode=0 Oct 30 13:18:17.457308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.457038 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" Oct 30 13:18:17.457526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.457056 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" event={"ID":"e5879660-fa09-42c2-ba35-6ef6a6c67930","Type":"ContainerDied","Data":"c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a"} Oct 30 13:18:17.457623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.457580 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql" event={"ID":"e5879660-fa09-42c2-ba35-6ef6a6c67930","Type":"ContainerDied","Data":"fd6e9d3975592fa19f7416f63a39ecc803e90149b36183806fb41ac953d82ab9"} Oct 30 13:18:17.457623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.457618 5515 scope.go:122] "RemoveContainer" containerID="c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a" Oct 30 13:18:17.472863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.472803 5515 scope.go:122] "RemoveContainer" containerID="c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a" Oct 30 13:18:17.473242 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:17.473212 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a\": container with ID starting with c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a not found: ID does not exist" containerID="c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a" Oct 30 13:18:17.473386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.473253 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a"} err="failed to get container status \"c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a\": rpc error: code = NotFound desc = could not find container \"c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a\": container with ID starting with c4497457dd4c5f8434a947fa4c88358d7bad5f9d568fbcf4b93ed77728adb08a not found: ID does not exist" Oct 30 13:18:17.523780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.523731 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql"] Oct 30 13:18:17.527855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.527824 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-b9mql"] Oct 30 13:18:17.627000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.626960 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:17.853889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:17.853793 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:18.002133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:18.002099 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" path="/var/lib/kubelet/pods/e5879660-fa09-42c2-ba35-6ef6a6c67930/volumes" Oct 30 13:18:19.305594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:19.305547 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:19.900411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:19.900381 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:18:20.020162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020131 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-token\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-token\") pod \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " Oct 30 13:18:20.020379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020197 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-certs\") pod \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " Oct 30 13:18:20.020379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020300 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istiod-ca-cert\") pod \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " Oct 30 13:18:20.020379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020351 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-data\") pod \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " Oct 30 13:18:20.020575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020395 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-podinfo\") pod \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " Oct 30 13:18:20.020575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020444 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-credential-socket\") pod \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " Oct 30 13:18:20.020575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020517 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-socket\") pod \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " Oct 30 13:18:20.020575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020551 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-kube-api-access-vdhpj\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-kube-api-access-vdhpj\") pod \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " Oct 30 13:18:20.020785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020598 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-envoy\") pod \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\" (UID: \"ac38c16d-786e-42c2-ae5e-42c2a2bef0da\") " Oct 30 13:18:20.020960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020567 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-certs" pod "ac38c16d-786e-42c2-ae5e-42c2a2bef0da" (UID: "ac38c16d-786e-42c2-ae5e-42c2a2bef0da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:20.020960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020660 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-data" pod "ac38c16d-786e-42c2-ae5e-42c2a2bef0da" (UID: "ac38c16d-786e-42c2-ae5e-42c2a2bef0da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:20.021097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020763 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istiod-ca-cert" pod "ac38c16d-786e-42c2-ae5e-42c2a2bef0da" (UID: "ac38c16d-786e-42c2-ae5e-42c2a2bef0da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:20.021097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020767 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-credential-socket" pod "ac38c16d-786e-42c2-ae5e-42c2a2bef0da" (UID: "ac38c16d-786e-42c2-ae5e-42c2a2bef0da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:20.021097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.020904 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-socket" pod "ac38c16d-786e-42c2-ae5e-42c2a2bef0da" (UID: "ac38c16d-786e-42c2-ae5e-42c2a2bef0da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:20.021304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.021258 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:20.021371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.021309 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:20.021371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.021323 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:20.021371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.021336 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:20.021371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.021351 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:20.022635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.022602 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-token" pod "ac38c16d-786e-42c2-ae5e-42c2a2bef0da" (UID: "ac38c16d-786e-42c2-ae5e-42c2a2bef0da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:20.022755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.022729 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-kube-api-access-vdhpj" pod "ac38c16d-786e-42c2-ae5e-42c2a2bef0da" (UID: "ac38c16d-786e-42c2-ae5e-42c2a2bef0da"). InnerVolumeSpecName "kube-api-access-vdhpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:20.022989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.022968 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-podinfo" pod "ac38c16d-786e-42c2-ae5e-42c2a2bef0da" (UID: "ac38c16d-786e-42c2-ae5e-42c2a2bef0da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:20.023025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.022990 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-envoy" pod "ac38c16d-786e-42c2-ae5e-42c2a2bef0da" (UID: "ac38c16d-786e-42c2-ae5e-42c2a2bef0da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:20.122677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.122638 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:20.122677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.122676 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:20.122907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.122689 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdhpj\" (UniqueName: \"kubernetes.io/projected/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-kube-api-access-vdhpj\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:20.122907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.122702 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac38c16d-786e-42c2-ae5e-42c2a2bef0da-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:20.479574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.479542 5515 generic.go:356] "Generic (PLEG): container finished" podID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerID="6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf" exitCode=0 Oct 30 13:18:20.480012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.479590 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" Oct 30 13:18:20.480012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.479626 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" event={"ID":"ac38c16d-786e-42c2-ae5e-42c2a2bef0da","Type":"ContainerDied","Data":"6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf"} Oct 30 13:18:20.480012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.479679 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5" event={"ID":"ac38c16d-786e-42c2-ae5e-42c2a2bef0da","Type":"ContainerDied","Data":"553db0fc1604dafa6129a9f91f14985a6c8f8ce9decf4e25551561891c9f8b99"} Oct 30 13:18:20.480012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.479701 5515 scope.go:122] "RemoveContainer" containerID="6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf" Oct 30 13:18:20.490866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.490840 5515 scope.go:122] "RemoveContainer" containerID="6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf" Oct 30 13:18:20.491154 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:20.491129 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf\": container with ID starting with 6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf not found: ID does not exist" containerID="6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf" Oct 30 13:18:20.491209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.491164 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf"} err="failed to get container status \"6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf\": rpc error: code = NotFound desc = could not find container \"6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf\": container with ID starting with 6bf2c90fcc414fea6c1d45f1b6264d9a387e32c07479a4b0cf15a49cb4aac0bf not found: ID does not exist" Oct 30 13:18:20.536537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.536503 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5"] Oct 30 13:18:20.549388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.549350 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57qt5"] Oct 30 13:18:20.999449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:20.999397 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:21.116045 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116007 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11526-26aa70"] Oct 30 13:18:21.116742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116722 5515 cpu_manager.go:414] "containerMap: removing container" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerName="cleanup" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116745 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerName="cleanup" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116757 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerName="cleanup" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116762 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerName="cleanup" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116769 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerName="cleanup" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116775 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerName="cleanup" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116784 5515 cpu_manager.go:414] "containerMap: removing container" podUID="87b2d2b8-78d4-4b35-889d-02ebe0c97fd2" containerName="payload-pre-processing" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116789 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b2d2b8-78d4-4b35-889d-02ebe0c97fd2" containerName="payload-pre-processing" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116800 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116805 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" Oct 30 13:18:21.116814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116818 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116824 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116837 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116842 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116855 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116860 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116872 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116877 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116884 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116889 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116898 5515 cpu_manager.go:414] "containerMap: removing container" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116903 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116911 5515 cpu_manager.go:414] "containerMap: removing container" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116915 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116921 5515 cpu_manager.go:414] "containerMap: removing container" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116925 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116934 5515 cpu_manager.go:414] "containerMap: removing container" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116941 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116951 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116956 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116964 5515 cpu_manager.go:414] "containerMap: removing container" podUID="18aaf89b-5970-46e1-b575-bc01f7b96862" containerName="maas-api" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116968 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18aaf89b-5970-46e1-b575-bc01f7b96862" containerName="maas-api" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116976 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116980 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116989 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0c75c816-7912-4460-b898-ad9da761027b" containerName="maas-api" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.116993 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c75c816-7912-4460-b898-ad9da761027b" containerName="maas-api" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117078 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5879660-fa09-42c2-ba35-6ef6a6c67930" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117089 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117094 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="18aaf89b-5970-46e1-b575-bc01f7b96862" containerName="maas-api" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117100 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117105 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c75c816-7912-4460-b898-ad9da761027b" containerName="maas-api" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117111 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117116 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117121 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117127 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b2d2b8-78d4-4b35-889d-02ebe0c97fd2" containerName="payload-pre-processing" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117133 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1018d8-6578-406e-a683-3c183fb963d0" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117138 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117143 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerName="cleanup" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117151 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab051748-23bd-4b4d-8b7a-dab99c1b8ec2" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117158 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" containerName="istio-proxy" Oct 30 13:18:21.117149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117165 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c93a624-1411-42ef-a6b6-5a3dfba23243" containerName="istio-proxy" Oct 30 13:18:21.118601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117170 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff0adac-2f41-4528-ab90-ff59a93db902" containerName="istio-proxy" Oct 30 13:18:21.118601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117176 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="347bcefa-34a9-4bc4-a5a6-c18fbfdf73e8" containerName="istio-proxy" Oct 30 13:18:21.118601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117182 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerName="cleanup" Oct 30 13:18:21.118601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.117187 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8735b619-6f32-41db-b5e3-9cf19c6e8a21" containerName="cleanup" Oct 30 13:18:21.122209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.122188 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-26aa70" Oct 30 13:18:21.135579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.135548 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11526-26aa70"] Oct 30 13:18:21.135980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.135956 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pdr\" (UniqueName: \"kubernetes.io/projected/6dfc1648-421f-41ac-b2de-1b6c030400c0-kube-api-access-85pdr\") pod \"test-curl-11526-26aa70\" (UID: \"6dfc1648-421f-41ac-b2de-1b6c030400c0\") " pod="opendatahub/test-curl-11526-26aa70" Oct 30 13:18:21.137245 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.137219 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ljkrn\"" Oct 30 13:18:21.237644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.237606 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pdr\" (UniqueName: \"kubernetes.io/projected/6dfc1648-421f-41ac-b2de-1b6c030400c0-kube-api-access-85pdr\") pod \"test-curl-11526-26aa70\" (UID: \"6dfc1648-421f-41ac-b2de-1b6c030400c0\") " pod="opendatahub/test-curl-11526-26aa70" Oct 30 13:18:21.262154 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.262080 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pdr\" (UniqueName: \"kubernetes.io/projected/6dfc1648-421f-41ac-b2de-1b6c030400c0-kube-api-access-85pdr\") pod \"test-curl-11526-26aa70\" (UID: \"6dfc1648-421f-41ac-b2de-1b6c030400c0\") " pod="opendatahub/test-curl-11526-26aa70" Oct 30 13:18:21.435670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.435633 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-26aa70" Oct 30 13:18:21.735796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:21.735758 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11526-26aa70"] Oct 30 13:18:21.737622 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:18:21.737592 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dfc1648_421f_41ac_b2de_1b6c030400c0.slice/crio-3fd2cad230f0e0f8cfdd2697f9b1b51eee79a3c255e3b78fa9fb7eb454ab3c9c WatchSource:0}: Error finding container 3fd2cad230f0e0f8cfdd2697f9b1b51eee79a3c255e3b78fa9fb7eb454ab3c9c: Status 404 returned error can't find the container with id 3fd2cad230f0e0f8cfdd2697f9b1b51eee79a3c255e3b78fa9fb7eb454ab3c9c Oct 30 13:18:22.001992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:22.001916 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac38c16d-786e-42c2-ae5e-42c2a2bef0da" path="/var/lib/kubelet/pods/ac38c16d-786e-42c2-ae5e-42c2a2bef0da/volumes" Oct 30 13:18:22.496035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:22.496000 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11526-26aa70" event={"ID":"6dfc1648-421f-41ac-b2de-1b6c030400c0","Type":"ContainerStarted","Data":"4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb"} Oct 30 13:18:22.496370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:22.496048 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11526-26aa70" event={"ID":"6dfc1648-421f-41ac-b2de-1b6c030400c0","Type":"ContainerStarted","Data":"3fd2cad230f0e0f8cfdd2697f9b1b51eee79a3c255e3b78fa9fb7eb454ab3c9c"} Oct 30 13:18:22.815037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:22.814954 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:23.478112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:23.478017 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11526-26aa70" podStartSLOduration=2.4779952720000002 podStartE2EDuration="2.477995272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:22.552381161 +0000 UTC m=+2720.679182058" watchObservedRunningTime="2026-10-30 13:18:23.477995272 +0000 UTC m=+2721.604796170" Oct 30 13:18:23.480261 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:23.480223 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11526-26aa70"] Oct 30 13:18:23.492532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:23.492495 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11526-26aa70"] Oct 30 13:18:24.509026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:24.508966 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11526-26aa70" podUID="6dfc1648-421f-41ac-b2de-1b6c030400c0" containerName="test-curl-11526-26aa70" containerID="cri-o://4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb" gracePeriod=2 Oct 30 13:18:24.523746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:24.523719 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:26.279211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.279177 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:26.712874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.712848 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-26aa70" Oct 30 13:18:26.820005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.819920 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfc1648-421f-41ac-b2de-1b6c030400c0-kube-api-access-85pdr\" (UniqueName: \"kubernetes.io/projected/6dfc1648-421f-41ac-b2de-1b6c030400c0-kube-api-access-85pdr\") pod \"6dfc1648-421f-41ac-b2de-1b6c030400c0\" (UID: \"6dfc1648-421f-41ac-b2de-1b6c030400c0\") " Oct 30 13:18:26.822447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.822401 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfc1648-421f-41ac-b2de-1b6c030400c0-kube-api-access-85pdr" pod "6dfc1648-421f-41ac-b2de-1b6c030400c0" (UID: "6dfc1648-421f-41ac-b2de-1b6c030400c0"). InnerVolumeSpecName "kube-api-access-85pdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:26.823375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.823353 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt"] Oct 30 13:18:26.824007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.823991 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerName="cleanup" Oct 30 13:18:26.824046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.824008 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerName="cleanup" Oct 30 13:18:26.824091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.824045 5515 cpu_manager.go:414] "containerMap: removing container" podUID="6dfc1648-421f-41ac-b2de-1b6c030400c0" containerName="test-curl-11526-26aa70" Oct 30 13:18:26.824091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.824051 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dfc1648-421f-41ac-b2de-1b6c030400c0" containerName="test-curl-11526-26aa70" Oct 30 13:18:26.824091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.824081 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerName="cleanup" Oct 30 13:18:26.824091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.824086 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf79802-820b-4a8a-b5d6-e554632f09b4" containerName="cleanup" Oct 30 13:18:26.824226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.824187 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dfc1648-421f-41ac-b2de-1b6c030400c0" containerName="test-curl-11526-26aa70" Oct 30 13:18:26.824226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.824201 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfef362-1a1a-417f-8d53-6573b67e2130" containerName="cleanup" Oct 30 13:18:26.827732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.827708 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:26.856529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.856501 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f1af9c\"/\"openshift-service-ca.crt\"" Oct 30 13:18:26.856670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.856595 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f1af9c\"/\"noten-model-f1af9c-kserve-self-signed-certs\"" Oct 30 13:18:26.856670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.856658 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f1af9c\"/\"kube-root-ca.crt\"" Oct 30 13:18:26.856814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.856802 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f1af9c\"/\"default-dockercfg-22b62\"" Oct 30 13:18:26.863262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.863232 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt"] Oct 30 13:18:26.921651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.921615 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-home\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:26.921846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.921665 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-model-cache\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:26.921846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.921782 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-tmp-dir\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:26.921846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.921826 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22pwz\" (UniqueName: \"kubernetes.io/projected/834a3e3b-7903-4130-bc62-6f90821e338b-kube-api-access-22pwz\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:26.921994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.921854 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-dshm\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:26.921994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.921962 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834a3e3b-7903-4130-bc62-6f90821e338b-tls-certs\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:26.922221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.922203 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85pdr\" (UniqueName: \"kubernetes.io/projected/6dfc1648-421f-41ac-b2de-1b6c030400c0-kube-api-access-85pdr\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:26.985546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:26.985513 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:27.023900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.023868 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-home\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.023900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.023905 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-model-cache\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.024183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.024038 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-tmp-dir\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.024183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.024109 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22pwz\" (UniqueName: \"kubernetes.io/projected/834a3e3b-7903-4130-bc62-6f90821e338b-kube-api-access-22pwz\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.024183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.024154 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-dshm\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.024387 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.024195 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834a3e3b-7903-4130-bc62-6f90821e338b-tls-certs\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.024387 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.024319 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-model-cache\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.024474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.024402 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-home\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.024474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.024446 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-tmp-dir\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.026502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.026480 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-dshm\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.026773 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.026756 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834a3e3b-7903-4130-bc62-6f90821e338b-tls-certs\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.087340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.087031 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22pwz\" (UniqueName: \"kubernetes.io/projected/834a3e3b-7903-4130-bc62-6f90821e338b-kube-api-access-22pwz\") pod \"noten-model-f1af9c-kserve-68c8f9f665-jmhlt\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.156884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.156839 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:27.468573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.468544 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt"] Oct 30 13:18:27.470161 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:18:27.470134 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod834a3e3b_7903_4130_bc62_6f90821e338b.slice/crio-1c3919f8c22f6d7c2f749479f44679c202ea67b13053d9ac68336ac489fb4cf4 WatchSource:0}: Error finding container 1c3919f8c22f6d7c2f749479f44679c202ea67b13053d9ac68336ac489fb4cf4: Status 404 returned error can't find the container with id 1c3919f8c22f6d7c2f749479f44679c202ea67b13053d9ac68336ac489fb4cf4 Oct 30 13:18:27.530964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.530934 5515 generic.go:356] "Generic (PLEG): container finished" podID="6dfc1648-421f-41ac-b2de-1b6c030400c0" containerID="4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb" exitCode=137 Oct 30 13:18:27.531141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.531000 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-26aa70" Oct 30 13:18:27.531141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.531027 5515 scope.go:122] "RemoveContainer" containerID="4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb" Oct 30 13:18:27.533965 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.533932 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" event={"ID":"834a3e3b-7903-4130-bc62-6f90821e338b","Type":"ContainerStarted","Data":"1c3919f8c22f6d7c2f749479f44679c202ea67b13053d9ac68336ac489fb4cf4"} Oct 30 13:18:27.543298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.543243 5515 scope.go:122] "RemoveContainer" containerID="4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb" Oct 30 13:18:27.543601 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:27.543582 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb\": container with ID starting with 4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb not found: ID does not exist" containerID="4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb" Oct 30 13:18:27.543668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.543612 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb"} err="failed to get container status \"4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb\": rpc error: code = NotFound desc = could not find container \"4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb\": container with ID starting with 4b2f0414a8571293f27a26484497bf534516c7fe551c0ee47b697ee4d31e9cfb not found: ID does not exist" Oct 30 13:18:27.555940 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:27.555902 5515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11526-26aa70\" is forbidden: User \"system:node:ip-10-0-134-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-193.ec2.internal' and this object" podUID="6dfc1648-421f-41ac-b2de-1b6c030400c0" pod="opendatahub/test-curl-11526-26aa70" Oct 30 13:18:27.904131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:27.904099 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:28.005681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:28.004011 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dfc1648-421f-41ac-b2de-1b6c030400c0" path="/var/lib/kubelet/pods/6dfc1648-421f-41ac-b2de-1b6c030400c0/volumes" Oct 30 13:18:28.433406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:28.433364 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8"] Oct 30 13:18:28.569418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:28.567251 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" event={"ID":"834a3e3b-7903-4130-bc62-6f90821e338b","Type":"ContainerStarted","Data":"690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1"} Oct 30 13:18:28.593913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:28.593862 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:28.996601 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:28.996559 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:18:29.157483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:29.157442 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:29.544422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:29.544385 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:31.290808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:31.290765 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:31.960842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:31.960784 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" podStartSLOduration=5.960767832 podStartE2EDuration="5.960767832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:28.62045712 +0000 UTC m=+2726.747258023" watchObservedRunningTime="2026-10-30 13:18:31.960767832 +0000 UTC m=+2730.087568729" Oct 30 13:18:31.964215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:31.964179 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt"] Oct 30 13:18:32.149072 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:32.146196 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:32.390291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:32.385784 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:32.642600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:32.642517 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:32.894838 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:32.894750 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:33.198964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:33.198876 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:34.310685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:34.310648 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk"] Oct 30 13:18:34.465495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:34.463021 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:34.847743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:34.847699 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:35.172842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.172792 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:35.560512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.560426 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp"] Oct 30 13:18:35.564943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.564915 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.593110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.593055 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-182377-openshift-default-dockercfg-j52qb\"" Oct 30 13:18:35.598587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.598555 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp"] Oct 30 13:18:35.606371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.606339 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsr8r\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-kube-api-access-tsr8r\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.606531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.606410 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f30d5c-865e-4525-a9dd-b5606c5ea789-istiod-ca-cert\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.606586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.606549 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-token\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.606640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.606613 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-credential-socket\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.606727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.606708 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-certs\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.606917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.606898 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-data\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.606976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.606960 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-envoy\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.607091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.607067 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-socket\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.607175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.607160 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-podinfo\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.708525 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.708491 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-podinfo\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.708732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.708552 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr8r\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-kube-api-access-tsr8r\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.708732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.708607 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f30d5c-865e-4525-a9dd-b5606c5ea789-istiod-ca-cert\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.708732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.708666 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-token\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.708732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.708712 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-credential-socket\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.709009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.708744 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-certs\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.709009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.708860 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-data\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.709009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.708913 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-envoy\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.709009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.708964 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-socket\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.709663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.709560 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-credential-socket\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.710092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.710047 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-socket\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.710092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.710074 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-certs\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.710319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.710114 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-data\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.710395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.710369 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f30d5c-865e-4525-a9dd-b5606c5ea789-istiod-ca-cert\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.711580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.711556 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-envoy\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.711730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.711713 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-podinfo\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.739424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.739353 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsr8r\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-kube-api-access-tsr8r\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.778356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.773558 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-token\") pod \"e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:35.787952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.785876 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:35.884699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:35.882847 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:36.015008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.014966 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:36.209100 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:18:36.209073 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f30d5c_865e_4525_a9dd_b5606c5ea789.slice/crio-6bd5d035b1ea75d7f5358e6a5f250319c9c9333955982ef4e729fcaf9e4186a0 WatchSource:0}: Error finding container 6bd5d035b1ea75d7f5358e6a5f250319c9c9333955982ef4e729fcaf9e4186a0: Status 404 returned error can't find the container with id 6bd5d035b1ea75d7f5358e6a5f250319c9c9333955982ef4e729fcaf9e4186a0 Oct 30 13:18:36.209299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.209246 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp"] Oct 30 13:18:36.212086 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.212040 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:36.212168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.212139 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:36.212212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.212181 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:36.635350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.632441 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" event={"ID":"10f30d5c-865e-4525-a9dd-b5606c5ea789","Type":"ContainerStarted","Data":"8036304a9e061c77360fae6bae2527df52f7058164145dbdd3c8ad0ca456f143"} Oct 30 13:18:36.635350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.632537 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" event={"ID":"10f30d5c-865e-4525-a9dd-b5606c5ea789","Type":"ContainerStarted","Data":"6bd5d035b1ea75d7f5358e6a5f250319c9c9333955982ef4e729fcaf9e4186a0"} Oct 30 13:18:36.691621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.691558 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" podStartSLOduration=1.6915386 podStartE2EDuration="1.6915386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:36.689883804 +0000 UTC m=+2734.816684702" watchObservedRunningTime="2026-10-30 13:18:36.6915386 +0000 UTC m=+2734.818339501" Oct 30 13:18:36.897350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.893548 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:36.921468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:36.921431 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:37.835953 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:37.835905 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:37.889772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:37.889741 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:38.377733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:38.377694 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:38.646237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:38.646209 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:38.647222 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:38.647200 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:18:38.783724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:38.783687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:39.034868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:39.034771 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:39.454607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:39.454576 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:39.675805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:39.675762 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:40.861860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:40.861824 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v"] Oct 30 13:18:40.862306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:40.862020 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" containerID="cri-o://56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15" gracePeriod=30 Oct 30 13:18:41.274738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.274705 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:41.433637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.433599 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg"] Oct 30 13:18:41.438425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.438406 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:41.458626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.458594 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg"] Oct 30 13:18:41.518291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.509188 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-182377-dockercfg-jsswm\"" Oct 30 13:18:41.518291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.509348 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-182377\"" Oct 30 13:18:41.550340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.544479 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4b920-aedd-4787-a8fa-876811fa43ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg\" (UID: \"b9a4b920-aedd-4787-a8fa-876811fa43ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:41.550340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.544538 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9pq\" (UniqueName: \"kubernetes.io/projected/b9a4b920-aedd-4787-a8fa-876811fa43ed-kube-api-access-qn9pq\") pod \"payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg\" (UID: \"b9a4b920-aedd-4787-a8fa-876811fa43ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:41.578446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.574146 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:41.646542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.646497 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4b920-aedd-4787-a8fa-876811fa43ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg\" (UID: \"b9a4b920-aedd-4787-a8fa-876811fa43ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:41.646766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.646559 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9pq\" (UniqueName: \"kubernetes.io/projected/b9a4b920-aedd-4787-a8fa-876811fa43ed-kube-api-access-qn9pq\") pod \"payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg\" (UID: \"b9a4b920-aedd-4787-a8fa-876811fa43ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:41.647342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.647256 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4b920-aedd-4787-a8fa-876811fa43ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg\" (UID: \"b9a4b920-aedd-4787-a8fa-876811fa43ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:41.670795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.670760 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9pq\" (UniqueName: \"kubernetes.io/projected/b9a4b920-aedd-4787-a8fa-876811fa43ed-kube-api-access-qn9pq\") pod \"payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg\" (UID: \"b9a4b920-aedd-4787-a8fa-876811fa43ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:41.753713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:41.753674 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:42.056971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:42.056943 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg"] Oct 30 13:18:42.058970 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:18:42.058909 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a4b920_aedd_4787_a8fa_876811fa43ed.slice/crio-988d764022c3e5405b99c06834749b1a94b3e2c6a157447002a52c16d985a3d7 WatchSource:0}: Error finding container 988d764022c3e5405b99c06834749b1a94b3e2c6a157447002a52c16d985a3d7: Status 404 returned error can't find the container with id 988d764022c3e5405b99c06834749b1a94b3e2c6a157447002a52c16d985a3d7 Oct 30 13:18:42.673850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:42.673814 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" event={"ID":"b9a4b920-aedd-4787-a8fa-876811fa43ed","Type":"ContainerStarted","Data":"aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2"} Oct 30 13:18:42.674053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:42.673883 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" event={"ID":"b9a4b920-aedd-4787-a8fa-876811fa43ed","Type":"ContainerStarted","Data":"988d764022c3e5405b99c06834749b1a94b3e2c6a157447002a52c16d985a3d7"} Oct 30 13:18:42.674053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:42.673926 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:42.722826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:42.722769 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" podStartSLOduration=1.722755915 podStartE2EDuration="1.722755915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:42.720001553 +0000 UTC m=+2740.846802452" watchObservedRunningTime="2026-10-30 13:18:42.722755915 +0000 UTC m=+2740.849556812" Oct 30 13:18:42.992974 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:42.992887 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks"] Oct 30 13:18:42.993171 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:42.993141 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" containerID="cri-o://96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763" gracePeriod=30 Oct 30 13:18:43.363342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.363240 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:43.456467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.456424 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" podUID="a3d10c25-9170-4c3c-9def-7bcfffc06d39" containerName="main" containerID="cri-o://c96d10066c8579c733353102014af05aa3042a4bab94a68941949bf9419dce41" gracePeriod=45 Oct 30 13:18:43.684826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.684792 5515 generic.go:356] "Generic (PLEG): container finished" podID="a3d10c25-9170-4c3c-9def-7bcfffc06d39" containerID="c96d10066c8579c733353102014af05aa3042a4bab94a68941949bf9419dce41" exitCode=0 Oct 30 13:18:43.685014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.684863 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" event={"ID":"a3d10c25-9170-4c3c-9def-7bcfffc06d39","Type":"ContainerDied","Data":"c96d10066c8579c733353102014af05aa3042a4bab94a68941949bf9419dce41"} Oct 30 13:18:43.745423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.745397 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:18:43.782380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.782343 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-home\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-home\") pod \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " Oct 30 13:18:43.782581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.782390 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-model-cache\") pod \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " Oct 30 13:18:43.782581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.782463 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tls-certs\") pod \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " Oct 30 13:18:43.782581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.782484 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d10c25-9170-4c3c-9def-7bcfffc06d39-kube-api-access-5vc5s\" (UniqueName: \"kubernetes.io/projected/a3d10c25-9170-4c3c-9def-7bcfffc06d39-kube-api-access-5vc5s\") pod \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " Oct 30 13:18:43.782581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.782543 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-dshm\") pod \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " Oct 30 13:18:43.782581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.782579 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tmp-dir\") pod \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\" (UID: \"a3d10c25-9170-4c3c-9def-7bcfffc06d39\") " Oct 30 13:18:43.782859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.782644 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-home" pod "a3d10c25-9170-4c3c-9def-7bcfffc06d39" (UID: "a3d10c25-9170-4c3c-9def-7bcfffc06d39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:43.782859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.782726 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-model-cache" pod "a3d10c25-9170-4c3c-9def-7bcfffc06d39" (UID: "a3d10c25-9170-4c3c-9def-7bcfffc06d39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:43.782966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.782888 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tmp-dir" pod "a3d10c25-9170-4c3c-9def-7bcfffc06d39" (UID: "a3d10c25-9170-4c3c-9def-7bcfffc06d39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:43.783392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.783362 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:43.783519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.783393 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:43.783519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.783410 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:43.784963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.784937 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tls-certs" pod "a3d10c25-9170-4c3c-9def-7bcfffc06d39" (UID: "a3d10c25-9170-4c3c-9def-7bcfffc06d39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:43.784963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.784944 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-dshm" pod "a3d10c25-9170-4c3c-9def-7bcfffc06d39" (UID: "a3d10c25-9170-4c3c-9def-7bcfffc06d39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:43.785115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.785018 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d10c25-9170-4c3c-9def-7bcfffc06d39-kube-api-access-5vc5s" pod "a3d10c25-9170-4c3c-9def-7bcfffc06d39" (UID: "a3d10c25-9170-4c3c-9def-7bcfffc06d39"). InnerVolumeSpecName "kube-api-access-5vc5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:43.884755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.884722 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d10c25-9170-4c3c-9def-7bcfffc06d39-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:43.884755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.884751 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vc5s\" (UniqueName: \"kubernetes.io/projected/a3d10c25-9170-4c3c-9def-7bcfffc06d39-kube-api-access-5vc5s\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:43.884755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.884760 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d10c25-9170-4c3c-9def-7bcfffc06d39-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:43.887137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:43.887100 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:44.123120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:44.122969 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:44.692865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:44.692832 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" event={"ID":"a3d10c25-9170-4c3c-9def-7bcfffc06d39","Type":"ContainerDied","Data":"b0319c431bd5a106834f0e5b7927efc4569ee9a8461c3296d0e0bba3f4f03d1d"} Oct 30 13:18:44.692865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:44.692853 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8" Oct 30 13:18:44.693437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:44.692886 5515 scope.go:122] "RemoveContainer" containerID="c96d10066c8579c733353102014af05aa3042a4bab94a68941949bf9419dce41" Oct 30 13:18:44.752192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:44.752144 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8"] Oct 30 13:18:44.763117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:44.763085 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7a8b38/auth-test-model-7a8b38-kserve-7dd78d4dfc-6trk8"] Oct 30 13:18:44.843897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:44.843857 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7cq2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:18:44.844061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:44.843918 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:18:45.558958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:45.558921 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:46.024125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.024092 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d10c25-9170-4c3c-9def-7bcfffc06d39" path="/var/lib/kubelet/pods/a3d10c25-9170-4c3c-9def-7bcfffc06d39/volumes" Oct 30 13:18:46.115864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.115829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:46.155191 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.155164 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 13:18:46.169237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.169202 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l"] Oct 30 13:18:46.169877 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.169860 5515 cpu_manager.go:414] "containerMap: removing container" podUID="a3d10c25-9170-4c3c-9def-7bcfffc06d39" containerName="main" Oct 30 13:18:46.169877 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.169878 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d10c25-9170-4c3c-9def-7bcfffc06d39" containerName="main" Oct 30 13:18:46.169982 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.169904 5515 cpu_manager.go:414] "containerMap: removing container" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" Oct 30 13:18:46.169982 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.169909 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" Oct 30 13:18:46.170050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.170029 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d10c25-9170-4c3c-9def-7bcfffc06d39" containerName="main" Oct 30 13:18:46.170050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.170042 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4053f95a-9bf2-4633-9016-219e780f7578" containerName="istio-proxy" Oct 30 13:18:46.173772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.173747 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:46.199775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.199731 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-451a21\"" Oct 30 13:18:46.200097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.200063 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-451a21-dockercfg-s72m4\"" Oct 30 13:18:46.203616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.203566 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l"] Oct 30 13:18:46.220645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.220611 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4053f95a-9bf2-4633-9016-219e780f7578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4053f95a-9bf2-4633-9016-219e780f7578-istio-podinfo\") pod \"4053f95a-9bf2-4633-9016-219e780f7578\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " Oct 30 13:18:46.220876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.220712 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-kube-api-access-z97wh\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-kube-api-access-z97wh\") pod \"4053f95a-9bf2-4633-9016-219e780f7578\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " Oct 30 13:18:46.220876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.220773 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-istio-token\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-istio-token\") pod \"4053f95a-9bf2-4633-9016-219e780f7578\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " Oct 30 13:18:46.220876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.220835 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-envoy\") pod \"4053f95a-9bf2-4633-9016-219e780f7578\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " Oct 30 13:18:46.220876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.220862 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4053f95a-9bf2-4633-9016-219e780f7578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4053f95a-9bf2-4633-9016-219e780f7578-istiod-ca-cert\") pod \"4053f95a-9bf2-4633-9016-219e780f7578\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " Oct 30 13:18:46.221343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.220929 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-socket\") pod \"4053f95a-9bf2-4633-9016-219e780f7578\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " Oct 30 13:18:46.221343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.220965 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-credential-socket\") pod \"4053f95a-9bf2-4633-9016-219e780f7578\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " Oct 30 13:18:46.221343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.221108 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-certs\") pod \"4053f95a-9bf2-4633-9016-219e780f7578\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " Oct 30 13:18:46.221343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.221147 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-data\") pod \"4053f95a-9bf2-4633-9016-219e780f7578\" (UID: \"4053f95a-9bf2-4633-9016-219e780f7578\") " Oct 30 13:18:46.221671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.221564 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zcl\" (UniqueName: \"kubernetes.io/projected/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-kube-api-access-n7zcl\") pod \"payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l\" (UID: \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:46.221671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.221663 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l\" (UID: \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:46.221829 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.221712 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4053f95a-9bf2-4633-9016-219e780f7578-istiod-ca-cert" pod "4053f95a-9bf2-4633-9016-219e780f7578" (UID: "4053f95a-9bf2-4633-9016-219e780f7578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:46.221905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.221875 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-credential-socket" pod "4053f95a-9bf2-4633-9016-219e780f7578" (UID: "4053f95a-9bf2-4633-9016-219e780f7578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:46.221984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.221961 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-data" pod "4053f95a-9bf2-4633-9016-219e780f7578" (UID: "4053f95a-9bf2-4633-9016-219e780f7578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:46.222059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.221975 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-socket" pod "4053f95a-9bf2-4633-9016-219e780f7578" (UID: "4053f95a-9bf2-4633-9016-219e780f7578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:46.222136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.222111 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-certs" pod "4053f95a-9bf2-4633-9016-219e780f7578" (UID: "4053f95a-9bf2-4633-9016-219e780f7578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:46.222205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.222188 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4053f95a-9bf2-4633-9016-219e780f7578-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.222295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.222213 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.222295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.222228 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.222295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.222242 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.224992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.224941 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-kube-api-access-z97wh" pod "4053f95a-9bf2-4633-9016-219e780f7578" (UID: "4053f95a-9bf2-4633-9016-219e780f7578"). InnerVolumeSpecName "kube-api-access-z97wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:46.226262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.225736 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-istio-token" pod "4053f95a-9bf2-4633-9016-219e780f7578" (UID: "4053f95a-9bf2-4633-9016-219e780f7578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:46.226262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.226036 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4053f95a-9bf2-4633-9016-219e780f7578-istio-podinfo" pod "4053f95a-9bf2-4633-9016-219e780f7578" (UID: "4053f95a-9bf2-4633-9016-219e780f7578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:46.227217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.227171 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-envoy" pod "4053f95a-9bf2-4633-9016-219e780f7578" (UID: "4053f95a-9bf2-4633-9016-219e780f7578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:46.264945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.264901 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x"] Oct 30 13:18:46.271692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.271659 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:46.289671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.289639 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x"] Oct 30 13:18:46.323243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.323205 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlh2g\" (UniqueName: \"kubernetes.io/projected/48085915-c753-4106-9bc8-faa198f108f9-kube-api-access-rlh2g\") pod \"payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x\" (UID: \"48085915-c753-4106-9bc8-faa198f108f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:46.323461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.323282 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48085915-c753-4106-9bc8-faa198f108f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x\" (UID: \"48085915-c753-4106-9bc8-faa198f108f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:46.323461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.323331 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zcl\" (UniqueName: \"kubernetes.io/projected/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-kube-api-access-n7zcl\") pod \"payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l\" (UID: \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:46.323461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.323434 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l\" (UID: \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:46.323642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.323612 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.323642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.323624 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.323642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.323635 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4053f95a-9bf2-4633-9016-219e780f7578-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.323866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.323643 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4053f95a-9bf2-4633-9016-219e780f7578-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.323866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.323656 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97wh\" (UniqueName: \"kubernetes.io/projected/4053f95a-9bf2-4633-9016-219e780f7578-kube-api-access-z97wh\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:46.324327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.324305 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l\" (UID: \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:46.368470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.365566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:46.400293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.390168 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zcl\" (UniqueName: \"kubernetes.io/projected/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-kube-api-access-n7zcl\") pod \"payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l\" (UID: \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:46.431290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.427555 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlh2g\" (UniqueName: \"kubernetes.io/projected/48085915-c753-4106-9bc8-faa198f108f9-kube-api-access-rlh2g\") pod \"payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x\" (UID: \"48085915-c753-4106-9bc8-faa198f108f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:46.431290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.427636 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48085915-c753-4106-9bc8-faa198f108f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x\" (UID: \"48085915-c753-4106-9bc8-faa198f108f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:46.431290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.428566 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48085915-c753-4106-9bc8-faa198f108f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x\" (UID: \"48085915-c753-4106-9bc8-faa198f108f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:46.456310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.455432 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlh2g\" (UniqueName: \"kubernetes.io/projected/48085915-c753-4106-9bc8-faa198f108f9-kube-api-access-rlh2g\") pod \"payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x\" (UID: \"48085915-c753-4106-9bc8-faa198f108f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:46.488906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.488874 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:46.588958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.588920 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:46.713822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.713786 5515 generic.go:356] "Generic (PLEG): container finished" podID="4053f95a-9bf2-4633-9016-219e780f7578" containerID="56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15" exitCode=0 Oct 30 13:18:46.714011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.713909 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" Oct 30 13:18:46.714011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.713916 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" event={"ID":"4053f95a-9bf2-4633-9016-219e780f7578","Type":"ContainerDied","Data":"56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15"} Oct 30 13:18:46.714011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.713977 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v" event={"ID":"4053f95a-9bf2-4633-9016-219e780f7578","Type":"ContainerDied","Data":"27415c5ee90e558adcaa41b56cee56737ab9a02968bad07a21aadea794aed370"} Oct 30 13:18:46.714198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.714009 5515 scope.go:122] "RemoveContainer" containerID="56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15" Oct 30 13:18:46.728913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.728889 5515 scope.go:122] "RemoveContainer" containerID="56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15" Oct 30 13:18:46.729247 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:46.729228 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15\": container with ID starting with 56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15 not found: ID does not exist" containerID="56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15" Oct 30 13:18:46.729352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.729254 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15"} err="failed to get container status \"56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15\": rpc error: code = NotFound desc = could not find container \"56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15\": container with ID starting with 56dea4d4121ac9b3ef13ba8811734acced107c2fb6c4f2567224d885ce91aa15 not found: ID does not exist" Oct 30 13:18:46.768029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.767999 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v"] Oct 30 13:18:46.780370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.780334 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7cq2v"] Oct 30 13:18:46.803854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.803828 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l"] Oct 30 13:18:46.805371 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:18:46.805348 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11abbb0d_84ba_4a30_b0c3_c3ba1b5d7575.slice/crio-28c4589fa9a93d5d6dab76e9c5c56110b08064523b99c6f2acb134a7698291b9 WatchSource:0}: Error finding container 28c4589fa9a93d5d6dab76e9c5c56110b08064523b99c6f2acb134a7698291b9: Status 404 returned error can't find the container with id 28c4589fa9a93d5d6dab76e9c5c56110b08064523b99c6f2acb134a7698291b9 Oct 30 13:18:46.899780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.899748 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x"] Oct 30 13:18:46.904732 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:18:46.904707 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48085915_c753_4106_9bc8_faa198f108f9.slice/crio-484fadb5ce046a7b8434f84a9f21b85a136d3dcac7a43832efa1b440bc718d1b WatchSource:0}: Error finding container 484fadb5ce046a7b8434f84a9f21b85a136d3dcac7a43832efa1b440bc718d1b: Status 404 returned error can't find the container with id 484fadb5ce046a7b8434f84a9f21b85a136d3dcac7a43832efa1b440bc718d1b Oct 30 13:18:46.986338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:46.986298 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" podUID="834a3e3b-7903-4130-bc62-6f90821e338b" containerName="main" containerID="cri-o://690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1" gracePeriod=45 Oct 30 13:18:47.270846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.270820 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:47.337947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.337909 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-tmp-dir\") pod \"834a3e3b-7903-4130-bc62-6f90821e338b\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " Oct 30 13:18:47.338126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.338032 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-home\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-home\") pod \"834a3e3b-7903-4130-bc62-6f90821e338b\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " Oct 30 13:18:47.338126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.338065 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-model-cache\") pod \"834a3e3b-7903-4130-bc62-6f90821e338b\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " Oct 30 13:18:47.338126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.338101 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-dshm\") pod \"834a3e3b-7903-4130-bc62-6f90821e338b\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " Oct 30 13:18:47.338485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.338192 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/834a3e3b-7903-4130-bc62-6f90821e338b-kube-api-access-22pwz\" (UniqueName: \"kubernetes.io/projected/834a3e3b-7903-4130-bc62-6f90821e338b-kube-api-access-22pwz\") pod \"834a3e3b-7903-4130-bc62-6f90821e338b\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " Oct 30 13:18:47.338485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.338221 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-tmp-dir" pod "834a3e3b-7903-4130-bc62-6f90821e338b" (UID: "834a3e3b-7903-4130-bc62-6f90821e338b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:47.338485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.338299 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/834a3e3b-7903-4130-bc62-6f90821e338b-tls-certs\" (UniqueName: \"kubernetes.io/secret/834a3e3b-7903-4130-bc62-6f90821e338b-tls-certs\") pod \"834a3e3b-7903-4130-bc62-6f90821e338b\" (UID: \"834a3e3b-7903-4130-bc62-6f90821e338b\") " Oct 30 13:18:47.338485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.338303 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-home" pod "834a3e3b-7903-4130-bc62-6f90821e338b" (UID: "834a3e3b-7903-4130-bc62-6f90821e338b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:47.338734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.338682 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-model-cache" pod "834a3e3b-7903-4130-bc62-6f90821e338b" (UID: "834a3e3b-7903-4130-bc62-6f90821e338b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:47.339109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.338989 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:47.339109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.339015 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:47.339109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.339029 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:47.340832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.340806 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834a3e3b-7903-4130-bc62-6f90821e338b-kube-api-access-22pwz" pod "834a3e3b-7903-4130-bc62-6f90821e338b" (UID: "834a3e3b-7903-4130-bc62-6f90821e338b"). InnerVolumeSpecName "kube-api-access-22pwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:47.351682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.351654 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-dshm" pod "834a3e3b-7903-4130-bc62-6f90821e338b" (UID: "834a3e3b-7903-4130-bc62-6f90821e338b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:47.351811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.351704 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834a3e3b-7903-4130-bc62-6f90821e338b-tls-certs" pod "834a3e3b-7903-4130-bc62-6f90821e338b" (UID: "834a3e3b-7903-4130-bc62-6f90821e338b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:47.440303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.440187 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a3e3b-7903-4130-bc62-6f90821e338b-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:47.440303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.440217 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22pwz\" (UniqueName: \"kubernetes.io/projected/834a3e3b-7903-4130-bc62-6f90821e338b-kube-api-access-22pwz\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:47.440303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.440228 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834a3e3b-7903-4130-bc62-6f90821e338b-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:47.501693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.501657 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:47.730566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.730477 5515 generic.go:356] "Generic (PLEG): container finished" podID="834a3e3b-7903-4130-bc62-6f90821e338b" containerID="690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1" exitCode=0 Oct 30 13:18:47.730726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.730564 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" Oct 30 13:18:47.730726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.730568 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" event={"ID":"834a3e3b-7903-4130-bc62-6f90821e338b","Type":"ContainerDied","Data":"690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1"} Oct 30 13:18:47.730726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.730620 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt" event={"ID":"834a3e3b-7903-4130-bc62-6f90821e338b","Type":"ContainerDied","Data":"1c3919f8c22f6d7c2f749479f44679c202ea67b13053d9ac68336ac489fb4cf4"} Oct 30 13:18:47.730726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.730703 5515 scope.go:122] "RemoveContainer" containerID="690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1" Oct 30 13:18:47.732548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.732518 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" event={"ID":"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575","Type":"ContainerStarted","Data":"03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce"} Oct 30 13:18:47.732669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.732605 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" event={"ID":"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575","Type":"ContainerStarted","Data":"28c4589fa9a93d5d6dab76e9c5c56110b08064523b99c6f2acb134a7698291b9"} Oct 30 13:18:47.732725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.732672 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:47.735441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.735414 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" event={"ID":"48085915-c753-4106-9bc8-faa198f108f9","Type":"ContainerStarted","Data":"ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73"} Oct 30 13:18:47.735556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.735463 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" event={"ID":"48085915-c753-4106-9bc8-faa198f108f9","Type":"ContainerStarted","Data":"484fadb5ce046a7b8434f84a9f21b85a136d3dcac7a43832efa1b440bc718d1b"} Oct 30 13:18:47.735556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.735503 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:47.744030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.744007 5515 scope.go:122] "RemoveContainer" containerID="690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1" Oct 30 13:18:47.744455 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:47.744429 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1\": container with ID starting with 690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1 not found: ID does not exist" containerID="690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1" Oct 30 13:18:47.744560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.744466 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1"} err="failed to get container status \"690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1\": rpc error: code = NotFound desc = could not find container \"690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1\": container with ID starting with 690ee5f86018b67c4ffdab3a2bd1fcdb439dc92ffb560bc41b26b5cb364b2ab1 not found: ID does not exist" Oct 30 13:18:47.790497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.790443 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" podStartSLOduration=1.7904291940000001 podStartE2EDuration="1.790429194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:47.788157741 +0000 UTC m=+2745.914958638" watchObservedRunningTime="2026-10-30 13:18:47.790429194 +0000 UTC m=+2745.917230092" Oct 30 13:18:47.825362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.825320 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt"] Oct 30 13:18:47.834539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.834495 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f1af9c/noten-model-f1af9c-kserve-68c8f9f665-jmhlt"] Oct 30 13:18:47.875435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:47.875385 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" podStartSLOduration=1.8753700960000002 podStartE2EDuration="1.875370096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:47.872545615 +0000 UTC m=+2745.999346514" watchObservedRunningTime="2026-10-30 13:18:47.875370096 +0000 UTC m=+2746.002170993" Oct 30 13:18:48.024318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.024143 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4053f95a-9bf2-4633-9016-219e780f7578" path="/var/lib/kubelet/pods/4053f95a-9bf2-4633-9016-219e780f7578/volumes" Oct 30 13:18:48.030242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.025116 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="834a3e3b-7903-4130-bc62-6f90821e338b" path="/var/lib/kubelet/pods/834a3e3b-7903-4130-bc62-6f90821e338b/volumes" Oct 30 13:18:48.287800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.287734 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:18:48.353200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.353159 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-envoy\") pod \"cc530b2d-6082-47d9-b62a-09593698d456\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " Oct 30 13:18:48.353436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.353327 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-certs\") pod \"cc530b2d-6082-47d9-b62a-09593698d456\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " Oct 30 13:18:48.353523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.353435 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-socket\") pod \"cc530b2d-6082-47d9-b62a-09593698d456\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " Oct 30 13:18:48.353687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.353640 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-certs" pod "cc530b2d-6082-47d9-b62a-09593698d456" (UID: "cc530b2d-6082-47d9-b62a-09593698d456"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.353767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.353730 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-socket" pod "cc530b2d-6082-47d9-b62a-09593698d456" (UID: "cc530b2d-6082-47d9-b62a-09593698d456"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.353809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.353797 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-data\") pod \"cc530b2d-6082-47d9-b62a-09593698d456\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " Oct 30 13:18:48.353870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.353851 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-credential-socket\") pod \"cc530b2d-6082-47d9-b62a-09593698d456\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " Oct 30 13:18:48.353948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.353887 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-istio-token\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-istio-token\") pod \"cc530b2d-6082-47d9-b62a-09593698d456\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " Oct 30 13:18:48.354012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.353968 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc530b2d-6082-47d9-b62a-09593698d456-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc530b2d-6082-47d9-b62a-09593698d456-istiod-ca-cert\") pod \"cc530b2d-6082-47d9-b62a-09593698d456\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " Oct 30 13:18:48.354067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354050 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-kube-api-access-sv49h\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-kube-api-access-sv49h\") pod \"cc530b2d-6082-47d9-b62a-09593698d456\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " Oct 30 13:18:48.354124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354075 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-data" pod "cc530b2d-6082-47d9-b62a-09593698d456" (UID: "cc530b2d-6082-47d9-b62a-09593698d456"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.354124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354109 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-credential-socket" pod "cc530b2d-6082-47d9-b62a-09593698d456" (UID: "cc530b2d-6082-47d9-b62a-09593698d456"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.354239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354126 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc530b2d-6082-47d9-b62a-09593698d456-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc530b2d-6082-47d9-b62a-09593698d456-istio-podinfo\") pod \"cc530b2d-6082-47d9-b62a-09593698d456\" (UID: \"cc530b2d-6082-47d9-b62a-09593698d456\") " Oct 30 13:18:48.354637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354443 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc530b2d-6082-47d9-b62a-09593698d456-istiod-ca-cert" pod "cc530b2d-6082-47d9-b62a-09593698d456" (UID: "cc530b2d-6082-47d9-b62a-09593698d456"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:48.354869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354848 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.354951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354873 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc530b2d-6082-47d9-b62a-09593698d456-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.354951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354889 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.354951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354902 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.354951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.354915 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.357254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.357213 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-envoy" pod "cc530b2d-6082-47d9-b62a-09593698d456" (UID: "cc530b2d-6082-47d9-b62a-09593698d456"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:48.357510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.357429 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-istio-token" pod "cc530b2d-6082-47d9-b62a-09593698d456" (UID: "cc530b2d-6082-47d9-b62a-09593698d456"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:48.357711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.357682 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-kube-api-access-sv49h" pod "cc530b2d-6082-47d9-b62a-09593698d456" (UID: "cc530b2d-6082-47d9-b62a-09593698d456"). InnerVolumeSpecName "kube-api-access-sv49h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:48.357974 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.357956 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc530b2d-6082-47d9-b62a-09593698d456-istio-podinfo" pod "cc530b2d-6082-47d9-b62a-09593698d456" (UID: "cc530b2d-6082-47d9-b62a-09593698d456"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:48.455929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.455894 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc530b2d-6082-47d9-b62a-09593698d456-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.455929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.455925 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.455929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.455934 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv49h\" (UniqueName: \"kubernetes.io/projected/cc530b2d-6082-47d9-b62a-09593698d456-kube-api-access-sv49h\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.456190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.455943 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc530b2d-6082-47d9-b62a-09593698d456-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:48.745645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.745613 5515 generic.go:356] "Generic (PLEG): container finished" podID="cc530b2d-6082-47d9-b62a-09593698d456" containerID="96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763" exitCode=0 Oct 30 13:18:48.745824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.745663 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" Oct 30 13:18:48.745824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.745702 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" event={"ID":"cc530b2d-6082-47d9-b62a-09593698d456","Type":"ContainerDied","Data":"96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763"} Oct 30 13:18:48.745824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.745757 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks" event={"ID":"cc530b2d-6082-47d9-b62a-09593698d456","Type":"ContainerDied","Data":"9a51ea862c5b4c0d204d14b971822f9ce17bc554aee6c2d66b87b10a1363511d"} Oct 30 13:18:48.745824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.745782 5515 scope.go:122] "RemoveContainer" containerID="96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763" Oct 30 13:18:48.756244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.756228 5515 scope.go:122] "RemoveContainer" containerID="96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763" Oct 30 13:18:48.756539 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:48.756517 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763\": container with ID starting with 96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763 not found: ID does not exist" containerID="96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763" Oct 30 13:18:48.756586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.756548 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763"} err="failed to get container status \"96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763\": rpc error: code = NotFound desc = could not find container \"96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763\": container with ID starting with 96f1e56e4036bf330abd904229e6ba3d44cc07b91b6361ef1dfc44bf5057e763 not found: ID does not exist" Oct 30 13:18:48.803867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.803832 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks"] Oct 30 13:18:48.816400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:48.816369 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-5f4ks"] Oct 30 13:18:49.338845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.338672 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" podUID="360f2411-be3c-44bb-81ba-f1c83efd59da" containerName="main" containerID="cri-o://f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411" gracePeriod=45 Oct 30 13:18:49.348566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.348521 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" podUID="360f2411-be3c-44bb-81ba-f1c83efd59da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.175:8000/ready\": EOF" Oct 30 13:18:49.384322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.384253 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:49.634658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.634631 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:18:49.670905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.670874 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-tmp-dir\") pod \"360f2411-be3c-44bb-81ba-f1c83efd59da\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " Oct 30 13:18:49.671087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671030 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/360f2411-be3c-44bb-81ba-f1c83efd59da-tls-certs\" (UniqueName: \"kubernetes.io/secret/360f2411-be3c-44bb-81ba-f1c83efd59da-tls-certs\") pod \"360f2411-be3c-44bb-81ba-f1c83efd59da\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " Oct 30 13:18:49.671087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671064 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-home\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-home\") pod \"360f2411-be3c-44bb-81ba-f1c83efd59da\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " Oct 30 13:18:49.671212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671128 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360f2411-be3c-44bb-81ba-f1c83efd59da-kube-api-access-4wmm9\" (UniqueName: \"kubernetes.io/projected/360f2411-be3c-44bb-81ba-f1c83efd59da-kube-api-access-4wmm9\") pod \"360f2411-be3c-44bb-81ba-f1c83efd59da\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " Oct 30 13:18:49.671212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671197 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-model-cache\") pod \"360f2411-be3c-44bb-81ba-f1c83efd59da\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " Oct 30 13:18:49.671341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671202 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-tmp-dir" pod "360f2411-be3c-44bb-81ba-f1c83efd59da" (UID: "360f2411-be3c-44bb-81ba-f1c83efd59da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:49.671341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671229 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-dshm\") pod \"360f2411-be3c-44bb-81ba-f1c83efd59da\" (UID: \"360f2411-be3c-44bb-81ba-f1c83efd59da\") " Oct 30 13:18:49.671467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671443 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-home" pod "360f2411-be3c-44bb-81ba-f1c83efd59da" (UID: "360f2411-be3c-44bb-81ba-f1c83efd59da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:49.671467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671455 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-model-cache" pod "360f2411-be3c-44bb-81ba-f1c83efd59da" (UID: "360f2411-be3c-44bb-81ba-f1c83efd59da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:49.671838 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671807 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:49.671838 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671833 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:49.672002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.671846 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:49.673557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.673531 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-dshm" pod "360f2411-be3c-44bb-81ba-f1c83efd59da" (UID: "360f2411-be3c-44bb-81ba-f1c83efd59da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:49.673730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.673710 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360f2411-be3c-44bb-81ba-f1c83efd59da-kube-api-access-4wmm9" pod "360f2411-be3c-44bb-81ba-f1c83efd59da" (UID: "360f2411-be3c-44bb-81ba-f1c83efd59da"). InnerVolumeSpecName "kube-api-access-4wmm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:49.673825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.673748 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360f2411-be3c-44bb-81ba-f1c83efd59da-tls-certs" pod "360f2411-be3c-44bb-81ba-f1c83efd59da" (UID: "360f2411-be3c-44bb-81ba-f1c83efd59da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:49.753794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.753755 5515 generic.go:356] "Generic (PLEG): container finished" podID="360f2411-be3c-44bb-81ba-f1c83efd59da" containerID="f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411" exitCode=0 Oct 30 13:18:49.753794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.753795 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" Oct 30 13:18:49.754055 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.753840 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" event={"ID":"360f2411-be3c-44bb-81ba-f1c83efd59da","Type":"ContainerDied","Data":"f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411"} Oct 30 13:18:49.754055 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.753885 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk" event={"ID":"360f2411-be3c-44bb-81ba-f1c83efd59da","Type":"ContainerDied","Data":"31a0078b197aa867323d0493b3c5d4744c371a2d72f670944242b6978fef512e"} Oct 30 13:18:49.754055 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.753907 5515 scope.go:122] "RemoveContainer" containerID="f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411" Oct 30 13:18:49.763926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.763909 5515 scope.go:122] "RemoveContainer" containerID="f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411" Oct 30 13:18:49.764202 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:49.764184 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411\": container with ID starting with f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411 not found: ID does not exist" containerID="f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411" Oct 30 13:18:49.764252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.764211 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411"} err="failed to get container status \"f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411\": rpc error: code = NotFound desc = could not find container \"f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411\": container with ID starting with f4c9b69e0195745230120f3b7f82e61b90d9fa1c92cb0d7ebeb64e5f7a492411 not found: ID does not exist" Oct 30 13:18:49.773331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.773307 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wmm9\" (UniqueName: \"kubernetes.io/projected/360f2411-be3c-44bb-81ba-f1c83efd59da-kube-api-access-4wmm9\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:49.773331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.773329 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/360f2411-be3c-44bb-81ba-f1c83efd59da-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:49.773487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.773341 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/360f2411-be3c-44bb-81ba-f1c83efd59da-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:18:49.806419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.806389 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk"] Oct 30 13:18:49.810227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:49.810203 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-98dabb/auth-test-model-98dabb-kserve-74f4554c57-rwcbk"] Oct 30 13:18:50.002298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:50.002238 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360f2411-be3c-44bb-81ba-f1c83efd59da" path="/var/lib/kubelet/pods/360f2411-be3c-44bb-81ba-f1c83efd59da/volumes" Oct 30 13:18:50.002781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:50.002767 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc530b2d-6082-47d9-b62a-09593698d456" path="/var/lib/kubelet/pods/cc530b2d-6082-47d9-b62a-09593698d456/volumes" Oct 30 13:18:50.661114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:50.661071 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:18:51.166574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:51.166543 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:52.986168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:52.986132 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:53.688003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:53.687969 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:18:54.696212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:54.696167 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:54.921948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:54.921910 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:55.340091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.340046 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:55.804188 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.804154 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk"] Oct 30 13:18:55.804987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.804970 5515 cpu_manager.go:414] "containerMap: removing container" podUID="834a3e3b-7903-4130-bc62-6f90821e338b" containerName="main" Oct 30 13:18:55.804987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.804988 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="834a3e3b-7903-4130-bc62-6f90821e338b" containerName="main" Oct 30 13:18:55.805103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.805017 5515 cpu_manager.go:414] "containerMap: removing container" podUID="360f2411-be3c-44bb-81ba-f1c83efd59da" containerName="main" Oct 30 13:18:55.805103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.805024 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360f2411-be3c-44bb-81ba-f1c83efd59da" containerName="main" Oct 30 13:18:55.805103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.805048 5515 cpu_manager.go:414] "containerMap: removing container" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" Oct 30 13:18:55.805103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.805055 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" Oct 30 13:18:55.805329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.805163 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="834a3e3b-7903-4130-bc62-6f90821e338b" containerName="main" Oct 30 13:18:55.805329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.805176 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="360f2411-be3c-44bb-81ba-f1c83efd59da" containerName="main" Oct 30 13:18:55.805329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.805190 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc530b2d-6082-47d9-b62a-09593698d456" containerName="istio-proxy" Oct 30 13:18:55.810656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.810627 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.830535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.830487 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk"] Oct 30 13:18:55.841553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.841519 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-037a5a-openshift-default-dockercfg-knxhc\"" Oct 30 13:18:55.880868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.880827 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-socket\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.881046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.880945 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-podinfo\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.881046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.880986 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-credential-socket\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.881046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.881009 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75jbg\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-kube-api-access-75jbg\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.881189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.881165 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istiod-ca-cert\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.881227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.881204 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-envoy\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.881263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.881230 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-token\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.881355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.881325 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-certs\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.881394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.881354 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-data\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.970755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.970718 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:55.982118 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982088 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-podinfo\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982148 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-credential-socket\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982175 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75jbg\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-kube-api-access-75jbg\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982256 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istiod-ca-cert\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982314 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-envoy\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982336 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-token\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982403 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-certs\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982425 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-data\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982474 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-socket\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982616 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-credential-socket\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982838 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-socket\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.982973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982932 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-certs\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.983050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.982980 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-data\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.983394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.983373 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istiod-ca-cert\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.984761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.984740 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-envoy\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:55.984867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:55.984768 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-podinfo\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:56.028476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.020589 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75jbg\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-kube-api-access-75jbg\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:56.062068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.059679 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-token\") pod \"e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:56.135506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.135467 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:56.207128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.207077 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:56.468478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.468446 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk"] Oct 30 13:18:56.470596 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:18:56.470570 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe1827e_d5b2_4052_a34f_7552b7a6b4b2.slice/crio-e383b8fa9beaea3c2dfa89ee451ff4a1040ad7cd0f15d97b2915ebcd666368ee WatchSource:0}: Error finding container e383b8fa9beaea3c2dfa89ee451ff4a1040ad7cd0f15d97b2915ebcd666368ee: Status 404 returned error can't find the container with id e383b8fa9beaea3c2dfa89ee451ff4a1040ad7cd0f15d97b2915ebcd666368ee Oct 30 13:18:56.473450 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.473412 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:56.473547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.473484 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:56.473547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.473513 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:18:56.494932 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.494901 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:56.813399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.813310 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" event={"ID":"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2","Type":"ContainerStarted","Data":"916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61"} Oct 30 13:18:56.813399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.813377 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" event={"ID":"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2","Type":"ContainerStarted","Data":"e383b8fa9beaea3c2dfa89ee451ff4a1040ad7cd0f15d97b2915ebcd666368ee"} Oct 30 13:18:56.864843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:56.864789 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" podStartSLOduration=1.864773649 podStartE2EDuration="1.864773649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:56.860343241 +0000 UTC m=+2754.987144139" watchObservedRunningTime="2026-10-30 13:18:56.864773649 +0000 UTC m=+2754.991574546" Oct 30 13:18:57.135893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:57.135862 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:57.141626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:57.141599 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:57.820342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:57.820305 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:57.821458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:57.821436 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:18:58.236824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:58.236786 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:58.750473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:58.750443 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:18:58.750661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:58.750513 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:18:58.994637 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:18:58.994602 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:18:59.197870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:59.197832 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:59.445128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:59.445089 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:18:59.677557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:18:59.677522 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:00.473162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:00.473128 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:01.261540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.261503 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:01.263325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.263298 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz"] Oct 30 13:19:01.267755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.267735 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:01.295388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.295357 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz"] Oct 30 13:19:01.303963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.303936 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-037a5a-dockercfg-9rx8p\"" Oct 30 13:19:01.303963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.303937 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-037a5a\"" Oct 30 13:19:01.372319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.369319 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpp82\" (UniqueName: \"kubernetes.io/projected/bb072b8d-a769-4020-ac97-482fe2ee24f4-kube-api-access-rpp82\") pod \"payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz\" (UID: \"bb072b8d-a769-4020-ac97-482fe2ee24f4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:01.372319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.369421 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb072b8d-a769-4020-ac97-482fe2ee24f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz\" (UID: \"bb072b8d-a769-4020-ac97-482fe2ee24f4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:01.377409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.377367 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g"] Oct 30 13:19:01.392715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.392681 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:01.445569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.442086 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g"] Oct 30 13:19:01.480292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.470781 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482d2fa0-cf8d-42d7-b848-d45f375217f2-plugins-config-volume\") pod \"payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g\" (UID: \"482d2fa0-cf8d-42d7-b848-d45f375217f2\") " pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:01.480292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.470874 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpp82\" (UniqueName: \"kubernetes.io/projected/bb072b8d-a769-4020-ac97-482fe2ee24f4-kube-api-access-rpp82\") pod \"payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz\" (UID: \"bb072b8d-a769-4020-ac97-482fe2ee24f4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:01.480292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.470938 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb072b8d-a769-4020-ac97-482fe2ee24f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz\" (UID: \"bb072b8d-a769-4020-ac97-482fe2ee24f4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:01.480292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.471026 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxnxf\" (UniqueName: \"kubernetes.io/projected/482d2fa0-cf8d-42d7-b848-d45f375217f2-kube-api-access-wxnxf\") pod \"payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g\" (UID: \"482d2fa0-cf8d-42d7-b848-d45f375217f2\") " pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:01.480292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.472329 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb072b8d-a769-4020-ac97-482fe2ee24f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz\" (UID: \"bb072b8d-a769-4020-ac97-482fe2ee24f4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:01.496393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.496357 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpp82\" (UniqueName: \"kubernetes.io/projected/bb072b8d-a769-4020-ac97-482fe2ee24f4-kube-api-access-rpp82\") pod \"payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz\" (UID: \"bb072b8d-a769-4020-ac97-482fe2ee24f4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:01.569374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.569255 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:01.572783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.572753 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxnxf\" (UniqueName: \"kubernetes.io/projected/482d2fa0-cf8d-42d7-b848-d45f375217f2-kube-api-access-wxnxf\") pod \"payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g\" (UID: \"482d2fa0-cf8d-42d7-b848-d45f375217f2\") " pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:01.572921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.572859 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482d2fa0-cf8d-42d7-b848-d45f375217f2-plugins-config-volume\") pod \"payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g\" (UID: \"482d2fa0-cf8d-42d7-b848-d45f375217f2\") " pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:01.573584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.573565 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482d2fa0-cf8d-42d7-b848-d45f375217f2-plugins-config-volume\") pod \"payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g\" (UID: \"482d2fa0-cf8d-42d7-b848-d45f375217f2\") " pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:01.583089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.583068 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:01.596811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.596782 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxnxf\" (UniqueName: \"kubernetes.io/projected/482d2fa0-cf8d-42d7-b848-d45f375217f2-kube-api-access-wxnxf\") pod \"payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g\" (UID: \"482d2fa0-cf8d-42d7-b848-d45f375217f2\") " pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:01.776128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.776096 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:01.898697 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:01.898657 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz"] Oct 30 13:19:01.907898 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:19:01.907870 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb072b8d_a769_4020_ac97_482fe2ee24f4.slice/crio-1e72ccfafbec8d46d25d2c29630f9407bb8c37a584d9f5a7824fb09aabb41578 WatchSource:0}: Error finding container 1e72ccfafbec8d46d25d2c29630f9407bb8c37a584d9f5a7824fb09aabb41578: Status 404 returned error can't find the container with id 1e72ccfafbec8d46d25d2c29630f9407bb8c37a584d9f5a7824fb09aabb41578 Oct 30 13:19:02.083092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.083063 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g"] Oct 30 13:19:02.084415 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:19:02.084388 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482d2fa0_cf8d_42d7_b848_d45f375217f2.slice/crio-38408baacd1509bd865be60c3433cf41f75d0c9b2d37af1eb37b5fa9cbaf8050 WatchSource:0}: Error finding container 38408baacd1509bd865be60c3433cf41f75d0c9b2d37af1eb37b5fa9cbaf8050: Status 404 returned error can't find the container with id 38408baacd1509bd865be60c3433cf41f75d0c9b2d37af1eb37b5fa9cbaf8050 Oct 30 13:19:02.363573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.363483 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:02.871725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.871688 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" event={"ID":"482d2fa0-cf8d-42d7-b848-d45f375217f2","Type":"ContainerStarted","Data":"9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845"} Oct 30 13:19:02.872213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.871760 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:02.872213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.871790 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" event={"ID":"482d2fa0-cf8d-42d7-b848-d45f375217f2","Type":"ContainerStarted","Data":"38408baacd1509bd865be60c3433cf41f75d0c9b2d37af1eb37b5fa9cbaf8050"} Oct 30 13:19:02.873492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.873467 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" event={"ID":"bb072b8d-a769-4020-ac97-482fe2ee24f4","Type":"ContainerStarted","Data":"3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58"} Oct 30 13:19:02.873614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.873516 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" event={"ID":"bb072b8d-a769-4020-ac97-482fe2ee24f4","Type":"ContainerStarted","Data":"1e72ccfafbec8d46d25d2c29630f9407bb8c37a584d9f5a7824fb09aabb41578"} Oct 30 13:19:02.873614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.873554 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:02.914818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.914762 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" podStartSLOduration=1.9147478470000001 podStartE2EDuration="1.914747847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:02.912865818 +0000 UTC m=+2761.039666717" watchObservedRunningTime="2026-10-30 13:19:02.914747847 +0000 UTC m=+2761.041548745" Oct 30 13:19:02.955820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:02.955763 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" podStartSLOduration=1.955746985 podStartE2EDuration="1.955746985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:02.954154937 +0000 UTC m=+2761.080955851" watchObservedRunningTime="2026-10-30 13:19:02.955746985 +0000 UTC m=+2761.082547974" Oct 30 13:19:04.015020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:04.014984 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:05.924934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:05.924900 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:07.246229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.246191 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x"] Oct 30 13:19:07.252168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.252144 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.276674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.276633 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-182377-serving-cert\"" Oct 30 13:19:07.290599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.290566 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x"] Oct 30 13:19:07.369878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.369843 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-maas-api-tls\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.370059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.369885 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73f4270-5c4f-4ee4-9164-161143c52466-openshift-service-ca\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.370059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.370015 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-metrics-certs\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.370150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.370135 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c566q\" (UniqueName: \"kubernetes.io/projected/c73f4270-5c4f-4ee4-9164-161143c52466-kube-api-access-c566q\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.464488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.464454 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:07.471918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.471883 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-maas-api-tls\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.471918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.471930 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73f4270-5c4f-4ee4-9164-161143c52466-openshift-service-ca\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.472155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.472010 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-metrics-certs\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.472155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.472146 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c566q\" (UniqueName: \"kubernetes.io/projected/c73f4270-5c4f-4ee4-9164-161143c52466-kube-api-access-c566q\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.472816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.472793 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73f4270-5c4f-4ee4-9164-161143c52466-openshift-service-ca\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.474661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.474628 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-metrics-certs\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.474781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.474763 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-maas-api-tls\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.496117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.496087 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c566q\" (UniqueName: \"kubernetes.io/projected/c73f4270-5c4f-4ee4-9164-161143c52466-kube-api-access-c566q\") pod \"maas-api-e2e-explicit-182377-757f6f54-7r28x\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.568252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.568157 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:07.874251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.874225 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x"] Oct 30 13:19:07.876199 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:19:07.876170 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73f4270_5c4f_4ee4_9164_161143c52466.slice/crio-fc70ef8a47ec8ed7eb24edcf20cf790b4160cc2a47f7642c4d6f7e744946c66f WatchSource:0}: Error finding container fc70ef8a47ec8ed7eb24edcf20cf790b4160cc2a47f7642c4d6f7e744946c66f: Status 404 returned error can't find the container with id fc70ef8a47ec8ed7eb24edcf20cf790b4160cc2a47f7642c4d6f7e744946c66f Oct 30 13:19:07.877935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.877916 5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:19:07.910126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.910085 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" event={"ID":"c73f4270-5c4f-4ee4-9164-161143c52466","Type":"ContainerStarted","Data":"fc70ef8a47ec8ed7eb24edcf20cf790b4160cc2a47f7642c4d6f7e744946c66f"} Oct 30 13:19:07.918201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:07.918170 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:08.919885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:08.919849 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" event={"ID":"c73f4270-5c4f-4ee4-9164-161143c52466","Type":"ContainerStarted","Data":"f5a9ecd33dfe6ffcb6d6ec4c02451fc260c732eefdfcb1a3d890e1c2ba27d3e5"} Oct 30 13:19:08.920526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:08.919935 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:09.078752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:09.078713 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:09.320294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:09.320192 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:11.133885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:11.133854 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:11.664431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:11.664396 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.126851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:12.126759 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:12.661403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:12.661369 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:13.196952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:13.196915 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:13.570136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:13.570051 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:13.883628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:13.883600 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:19:13.884036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:13.883665 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:19:13.936232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:13.936172 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" podStartSLOduration=6.529163982 podStartE2EDuration="6.93615656s" totalImagesPullingTime="406.992578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:08.9739715 +0000 UTC m=+2767.100772399" watchObservedRunningTime="2026-10-30 13:19:13.93615656 +0000 UTC m=+2772.062957458" Oct 30 13:19:14.933435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:14.933354 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:19:15.162597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:15.162558 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:15.749422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:15.749386 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:17.089192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:17.089148 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:17.529579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:17.529540 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:19.170047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:19.169975 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:19.455348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:19.455247 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:20.895132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:20.895101 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:22.665092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:22.665050 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:24.329243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:24.329208 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:24.596381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:24.595330 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:19:26.270096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:26.270067 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:28.167833 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:28.167795 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq"] Oct 30 13:19:28.168427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:28.168045 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" containerID="cri-o://4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5" gracePeriod=30 Oct 30 13:19:28.186980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:28.186942 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:28.989939 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:19:28.989899 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:19:29.259690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:29.259576 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:29.827803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:29.827767 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:29.959290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:29.951097 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b"] Oct 30 13:19:29.959290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:29.951413 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" containerID="cri-o://044ead6e5ce305a04a0f95a5e9c0f188b34cc8969098fcd4582354a64086999d" gracePeriod=30 Oct 30 13:19:30.074166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:30.074125 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:30.475545 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:30.475502 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:31.946095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:31.946063 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:33.091648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.091609 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:33.334032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.334000 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:33.470701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.470671 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 13:19:33.500065 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500030 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-token\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-token\") pod \"52f163fa-56a6-411a-802b-d421d7b68e6e\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " Oct 30 13:19:33.500241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500150 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-credential-socket\") pod \"52f163fa-56a6-411a-802b-d421d7b68e6e\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " Oct 30 13:19:33.500241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500201 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-kube-api-access-2jwnk\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-kube-api-access-2jwnk\") pod \"52f163fa-56a6-411a-802b-d421d7b68e6e\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " Oct 30 13:19:33.500356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500330 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-podinfo\") pod \"52f163fa-56a6-411a-802b-d421d7b68e6e\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " Oct 30 13:19:33.500548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500528 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-data\") pod \"52f163fa-56a6-411a-802b-d421d7b68e6e\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " Oct 30 13:19:33.500815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500533 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-credential-socket" pod "52f163fa-56a6-411a-802b-d421d7b68e6e" (UID: "52f163fa-56a6-411a-802b-d421d7b68e6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:33.500967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500726 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-data" pod "52f163fa-56a6-411a-802b-d421d7b68e6e" (UID: "52f163fa-56a6-411a-802b-d421d7b68e6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:33.500967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500800 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-socket\") pod \"52f163fa-56a6-411a-802b-d421d7b68e6e\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " Oct 30 13:19:33.500967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500933 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-certs\") pod \"52f163fa-56a6-411a-802b-d421d7b68e6e\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " Oct 30 13:19:33.501149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500984 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-socket" pod "52f163fa-56a6-411a-802b-d421d7b68e6e" (UID: "52f163fa-56a6-411a-802b-d421d7b68e6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:33.501149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.500989 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-envoy\") pod \"52f163fa-56a6-411a-802b-d421d7b68e6e\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " Oct 30 13:19:33.501149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.501070 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f163fa-56a6-411a-802b-d421d7b68e6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f163fa-56a6-411a-802b-d421d7b68e6e-istiod-ca-cert\") pod \"52f163fa-56a6-411a-802b-d421d7b68e6e\" (UID: \"52f163fa-56a6-411a-802b-d421d7b68e6e\") " Oct 30 13:19:33.501442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.501417 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-certs" pod "52f163fa-56a6-411a-802b-d421d7b68e6e" (UID: "52f163fa-56a6-411a-802b-d421d7b68e6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:33.501709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.501689 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f163fa-56a6-411a-802b-d421d7b68e6e-istiod-ca-cert" pod "52f163fa-56a6-411a-802b-d421d7b68e6e" (UID: "52f163fa-56a6-411a-802b-d421d7b68e6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:33.501781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.501746 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:33.501781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.501766 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:33.501863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.501781 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:33.501863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.501793 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:33.502760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.502729 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-kube-api-access-2jwnk" pod "52f163fa-56a6-411a-802b-d421d7b68e6e" (UID: "52f163fa-56a6-411a-802b-d421d7b68e6e"). InnerVolumeSpecName "kube-api-access-2jwnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:33.502851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.502786 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-podinfo" pod "52f163fa-56a6-411a-802b-d421d7b68e6e" (UID: "52f163fa-56a6-411a-802b-d421d7b68e6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:19:33.502851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.502820 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-token" pod "52f163fa-56a6-411a-802b-d421d7b68e6e" (UID: "52f163fa-56a6-411a-802b-d421d7b68e6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:33.503578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.503558 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-envoy" pod "52f163fa-56a6-411a-802b-d421d7b68e6e" (UID: "52f163fa-56a6-411a-802b-d421d7b68e6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:33.603485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.603439 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jwnk\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-kube-api-access-2jwnk\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:33.603485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.603478 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:33.603485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.603488 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:33.603485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.603497 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f163fa-56a6-411a-802b-d421d7b68e6e-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:33.603784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:33.603506 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f163fa-56a6-411a-802b-d421d7b68e6e-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:34.218687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.218654 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:34.257213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.257179 5515 generic.go:356] "Generic (PLEG): container finished" podID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerID="4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5" exitCode=0 Oct 30 13:19:34.257425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.257245 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" Oct 30 13:19:34.257425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.257278 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" event={"ID":"52f163fa-56a6-411a-802b-d421d7b68e6e","Type":"ContainerDied","Data":"4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5"} Oct 30 13:19:34.257425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.257339 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq" event={"ID":"52f163fa-56a6-411a-802b-d421d7b68e6e","Type":"ContainerDied","Data":"843c845ac2fba0488644d63c4f5a397af986c7ced12a737112bbe106fa5755a0"} Oct 30 13:19:34.257425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.257367 5515 scope.go:122] "RemoveContainer" containerID="4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5" Oct 30 13:19:34.268460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.268438 5515 scope.go:122] "RemoveContainer" containerID="983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c" Oct 30 13:19:34.278813 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.278794 5515 scope.go:122] "RemoveContainer" containerID="4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5" Oct 30 13:19:34.279125 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:19:34.279104 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5\": container with ID starting with 4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5 not found: ID does not exist" containerID="4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5" Oct 30 13:19:34.279180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.279134 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5"} err="failed to get container status \"4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5\": rpc error: code = NotFound desc = could not find container \"4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5\": container with ID starting with 4467ac456d76adf535dab36beede23b6a0fafe9139590861663ddadf1462fda5 not found: ID does not exist" Oct 30 13:19:34.279180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.279151 5515 scope.go:122] "RemoveContainer" containerID="983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c" Oct 30 13:19:34.279466 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:19:34.279446 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c\": container with ID starting with 983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c not found: ID does not exist" containerID="983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c" Oct 30 13:19:34.279515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.279473 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c"} err="failed to get container status \"983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c\": rpc error: code = NotFound desc = could not find container \"983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c\": container with ID starting with 983744e4f2b0aab8ea99506200269b0a9a927152d418bf70c2286abb155f1f2c not found: ID does not exist" Oct 30 13:19:34.323214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.323182 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq"] Oct 30 13:19:34.335193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.335163 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-c9zzq"] Oct 30 13:19:34.801210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.801173 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf"] Oct 30 13:19:34.802348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.802327 5515 cpu_manager.go:414] "containerMap: removing container" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" Oct 30 13:19:34.802404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.802351 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" Oct 30 13:19:34.802557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.802542 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" Oct 30 13:19:34.806930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.806904 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:34.815414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.815384 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:34.825604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.825576 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf"] Oct 30 13:19:34.826535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.826440 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-63499e\"" Oct 30 13:19:34.854433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.854398 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr"] Oct 30 13:19:34.855208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.855187 5515 cpu_manager.go:414] "containerMap: removing container" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" Oct 30 13:19:34.855379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.855211 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" Oct 30 13:19:34.855486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.855469 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" containerName="istio-proxy" Oct 30 13:19:34.859351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.859326 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:34.876414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.876384 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr"] Oct 30 13:19:34.878798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.878773 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-63499e-dockercfg-tzlwt\"" Oct 30 13:19:34.923706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.921281 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbs4\" (UniqueName: \"kubernetes.io/projected/85074b34-4622-4d92-a481-2075120a066c-kube-api-access-qkbs4\") pod \"payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf\" (UID: \"85074b34-4622-4d92-a481-2075120a066c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:34.923706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.921379 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c02cc8-b166-4d7a-911d-1f731e97f461-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr\" (UID: \"f1c02cc8-b166-4d7a-911d-1f731e97f461\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:34.923706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.921430 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85074b34-4622-4d92-a481-2075120a066c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf\" (UID: \"85074b34-4622-4d92-a481-2075120a066c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:34.923706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:34.921467 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9hr8\" (UniqueName: \"kubernetes.io/projected/f1c02cc8-b166-4d7a-911d-1f731e97f461-kube-api-access-c9hr8\") pod \"payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr\" (UID: \"f1c02cc8-b166-4d7a-911d-1f731e97f461\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:35.025792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.024053 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbs4\" (UniqueName: \"kubernetes.io/projected/85074b34-4622-4d92-a481-2075120a066c-kube-api-access-qkbs4\") pod \"payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf\" (UID: \"85074b34-4622-4d92-a481-2075120a066c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:35.025792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.024163 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c02cc8-b166-4d7a-911d-1f731e97f461-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr\" (UID: \"f1c02cc8-b166-4d7a-911d-1f731e97f461\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:35.025792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.024214 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85074b34-4622-4d92-a481-2075120a066c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf\" (UID: \"85074b34-4622-4d92-a481-2075120a066c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:35.025792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.024252 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9hr8\" (UniqueName: \"kubernetes.io/projected/f1c02cc8-b166-4d7a-911d-1f731e97f461-kube-api-access-c9hr8\") pod \"payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr\" (UID: \"f1c02cc8-b166-4d7a-911d-1f731e97f461\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:35.026056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.025934 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c02cc8-b166-4d7a-911d-1f731e97f461-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr\" (UID: \"f1c02cc8-b166-4d7a-911d-1f731e97f461\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:35.026846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.026815 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85074b34-4622-4d92-a481-2075120a066c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf\" (UID: \"85074b34-4622-4d92-a481-2075120a066c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:35.043425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.043391 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:35.049035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.049005 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbs4\" (UniqueName: \"kubernetes.io/projected/85074b34-4622-4d92-a481-2075120a066c-kube-api-access-qkbs4\") pod \"payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf\" (UID: \"85074b34-4622-4d92-a481-2075120a066c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:35.049196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.049093 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9hr8\" (UniqueName: \"kubernetes.io/projected/f1c02cc8-b166-4d7a-911d-1f731e97f461-kube-api-access-c9hr8\") pod \"payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr\" (UID: \"f1c02cc8-b166-4d7a-911d-1f731e97f461\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:35.130218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.130134 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:35.181532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.181498 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:35.275415 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.275383 5515 generic.go:356] "Generic (PLEG): container finished" podID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerID="044ead6e5ce305a04a0f95a5e9c0f188b34cc8969098fcd4582354a64086999d" exitCode=0 Oct 30 13:19:35.275848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.275507 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" event={"ID":"ff42defd-792a-4658-8ac3-e7fef9e4d459","Type":"ContainerDied","Data":"044ead6e5ce305a04a0f95a5e9c0f188b34cc8969098fcd4582354a64086999d"} Oct 30 13:19:35.445769 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:19:35.445737 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85074b34_4622_4d92_a481_2075120a066c.slice/crio-1f3723220472d2d416da6150c8f4217eaa8c696f931c2d9083cf714cd05f8177 WatchSource:0}: Error finding container 1f3723220472d2d416da6150c8f4217eaa8c696f931c2d9083cf714cd05f8177: Status 404 returned error can't find the container with id 1f3723220472d2d416da6150c8f4217eaa8c696f931c2d9083cf714cd05f8177 Oct 30 13:19:35.446227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.446208 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf"] Oct 30 13:19:35.507150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.507125 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr"] Oct 30 13:19:35.512713 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:19:35.512684 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c02cc8_b166_4d7a_911d_1f731e97f461.slice/crio-fb81992713cef80a35bc853e3b1fa194be9bdc01aea4e84d95e1c91e3a801738 WatchSource:0}: Error finding container fb81992713cef80a35bc853e3b1fa194be9bdc01aea4e84d95e1c91e3a801738: Status 404 returned error can't find the container with id fb81992713cef80a35bc853e3b1fa194be9bdc01aea4e84d95e1c91e3a801738 Oct 30 13:19:35.702231 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.702163 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:19:35.737552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737520 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-kube-api-access-m6nmx\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-kube-api-access-m6nmx\") pod \"ff42defd-792a-4658-8ac3-e7fef9e4d459\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " Oct 30 13:19:35.737704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737569 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-data\") pod \"ff42defd-792a-4658-8ac3-e7fef9e4d459\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " Oct 30 13:19:35.737704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737628 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-token\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-token\") pod \"ff42defd-792a-4658-8ac3-e7fef9e4d459\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " Oct 30 13:19:35.737704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737700 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-credential-socket\") pod \"ff42defd-792a-4658-8ac3-e7fef9e4d459\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " Oct 30 13:19:35.737876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737743 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-envoy\") pod \"ff42defd-792a-4658-8ac3-e7fef9e4d459\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " Oct 30 13:19:35.737876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737794 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-podinfo\") pod \"ff42defd-792a-4658-8ac3-e7fef9e4d459\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " Oct 30 13:19:35.737984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737863 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-data" pod "ff42defd-792a-4658-8ac3-e7fef9e4d459" (UID: "ff42defd-792a-4658-8ac3-e7fef9e4d459"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:35.737984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737894 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff42defd-792a-4658-8ac3-e7fef9e4d459-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42defd-792a-4658-8ac3-e7fef9e4d459-istiod-ca-cert\") pod \"ff42defd-792a-4658-8ac3-e7fef9e4d459\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " Oct 30 13:19:35.737984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737934 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-certs\") pod \"ff42defd-792a-4658-8ac3-e7fef9e4d459\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " Oct 30 13:19:35.738151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.737986 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-credential-socket" pod "ff42defd-792a-4658-8ac3-e7fef9e4d459" (UID: "ff42defd-792a-4658-8ac3-e7fef9e4d459"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:35.738151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.738021 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-socket\") pod \"ff42defd-792a-4658-8ac3-e7fef9e4d459\" (UID: \"ff42defd-792a-4658-8ac3-e7fef9e4d459\") " Oct 30 13:19:35.738335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.738307 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-certs" pod "ff42defd-792a-4658-8ac3-e7fef9e4d459" (UID: "ff42defd-792a-4658-8ac3-e7fef9e4d459"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:35.738413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.738348 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff42defd-792a-4658-8ac3-e7fef9e4d459-istiod-ca-cert" pod "ff42defd-792a-4658-8ac3-e7fef9e4d459" (UID: "ff42defd-792a-4658-8ac3-e7fef9e4d459"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:35.738613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.738591 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-socket" pod "ff42defd-792a-4658-8ac3-e7fef9e4d459" (UID: "ff42defd-792a-4658-8ac3-e7fef9e4d459"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:35.738792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.738759 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:35.738851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.738801 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42defd-792a-4658-8ac3-e7fef9e4d459-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:35.738851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.738816 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:35.738851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.738829 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:35.738851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.738842 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:35.739908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.739888 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-token" pod "ff42defd-792a-4658-8ac3-e7fef9e4d459" (UID: "ff42defd-792a-4658-8ac3-e7fef9e4d459"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:35.740013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.739915 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-kube-api-access-m6nmx" pod "ff42defd-792a-4658-8ac3-e7fef9e4d459" (UID: "ff42defd-792a-4658-8ac3-e7fef9e4d459"). InnerVolumeSpecName "kube-api-access-m6nmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:35.740331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.740311 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-podinfo" pod "ff42defd-792a-4658-8ac3-e7fef9e4d459" (UID: "ff42defd-792a-4658-8ac3-e7fef9e4d459"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:19:35.740331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.740321 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-envoy" pod "ff42defd-792a-4658-8ac3-e7fef9e4d459" (UID: "ff42defd-792a-4658-8ac3-e7fef9e4d459"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:35.839935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.839901 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:35.839935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.839932 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:35.839935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.839942 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42defd-792a-4658-8ac3-e7fef9e4d459-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:35.841224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:35.839950 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6nmx\" (UniqueName: \"kubernetes.io/projected/ff42defd-792a-4658-8ac3-e7fef9e4d459-kube-api-access-m6nmx\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:36.020434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.020402 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f163fa-56a6-411a-802b-d421d7b68e6e" path="/var/lib/kubelet/pods/52f163fa-56a6-411a-802b-d421d7b68e6e/volumes" Oct 30 13:19:36.044811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.044741 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:36.283878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.283781 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" event={"ID":"85074b34-4622-4d92-a481-2075120a066c","Type":"ContainerStarted","Data":"a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140"} Oct 30 13:19:36.283878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.283855 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:36.283878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.283877 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" event={"ID":"85074b34-4622-4d92-a481-2075120a066c","Type":"ContainerStarted","Data":"1f3723220472d2d416da6150c8f4217eaa8c696f931c2d9083cf714cd05f8177"} Oct 30 13:19:36.285661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.285629 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" event={"ID":"f1c02cc8-b166-4d7a-911d-1f731e97f461","Type":"ContainerStarted","Data":"1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b"} Oct 30 13:19:36.285791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.285684 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" event={"ID":"f1c02cc8-b166-4d7a-911d-1f731e97f461","Type":"ContainerStarted","Data":"fb81992713cef80a35bc853e3b1fa194be9bdc01aea4e84d95e1c91e3a801738"} Oct 30 13:19:36.285791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.285721 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:36.287379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.287358 5515 generic.go:356] "Generic (PLEG): container finished" podID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerID="8036304a9e061c77360fae6bae2527df52f7058164145dbdd3c8ad0ca456f143" exitCode=0 Oct 30 13:19:36.287461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.287435 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" event={"ID":"10f30d5c-865e-4525-a9dd-b5606c5ea789","Type":"ContainerDied","Data":"8036304a9e061c77360fae6bae2527df52f7058164145dbdd3c8ad0ca456f143"} Oct 30 13:19:36.287820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.287803 5515 scope.go:122] "RemoveContainer" containerID="8036304a9e061c77360fae6bae2527df52f7058164145dbdd3c8ad0ca456f143" Oct 30 13:19:36.288860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.288839 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" event={"ID":"ff42defd-792a-4658-8ac3-e7fef9e4d459","Type":"ContainerDied","Data":"2147d56f17e940679015182d6f82fb809a536468ec29fabfd5df1482225197cb"} Oct 30 13:19:36.288957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.288868 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b" Oct 30 13:19:36.288957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.288883 5515 scope.go:122] "RemoveContainer" containerID="044ead6e5ce305a04a0f95a5e9c0f188b34cc8969098fcd4582354a64086999d" Oct 30 13:19:36.289626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.289593 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:36.289704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.289664 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:36.289767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.289705 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:36.328189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.328146 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" podStartSLOduration=2.328133573 podStartE2EDuration="2.328133573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:36.326759232 +0000 UTC m=+2794.453560131" watchObservedRunningTime="2026-10-30 13:19:36.328133573 +0000 UTC m=+2794.454934506" Oct 30 13:19:36.365386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.365359 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b"] Oct 30 13:19:36.370147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.370121 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-k4j4b"] Oct 30 13:19:36.460715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:36.460648 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" podStartSLOduration=2.460628059 podStartE2EDuration="2.460628059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:36.459260976 +0000 UTC m=+2794.586061876" watchObservedRunningTime="2026-10-30 13:19:36.460628059 +0000 UTC m=+2794.587428958" Oct 30 13:19:37.326279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:37.326222 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" event={"ID":"10f30d5c-865e-4525-a9dd-b5606c5ea789","Type":"ContainerStarted","Data":"58d877708fa97d374d847e88504e62e3ca7e880e6f819687d5be9ec8d15d44fc"} Oct 30 13:19:37.868376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:37.868337 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:37.883333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:37.883302 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:19:37.884735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:37.884709 5515 patch_prober.go:29] interesting pod/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 13:19:37.884850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:37.884763 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 13:19:38.030694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:38.030660 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" path="/var/lib/kubelet/pods/ff42defd-792a-4658-8ac3-e7fef9e4d459/volumes" Oct 30 13:19:38.646352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:38.646324 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:19:38.884521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:38.884485 5515 patch_prober.go:29] interesting pod/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 13:19:38.884716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:38.884547 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 13:19:39.718364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:39.713220 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:39.883520 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:39.883485 5515 patch_prober.go:29] interesting pod/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 13:19:39.883707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:39.883538 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 13:19:40.884286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:40.884236 5515 patch_prober.go:29] interesting pod/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 13:19:40.884689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:40.884354 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 13:19:41.120326 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:41.120167 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z"] Oct 30 13:19:41.121856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:41.121820 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" containerID="cri-o://97bd4bb657f18a83420b7d4502e837cb8b7a5a7fedfa7378a5f40cc08a841555" gracePeriod=30 Oct 30 13:19:41.574154 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:41.574119 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:41.883574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:41.883550 5515 patch_prober.go:29] interesting pod/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 13:19:41.883721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:41.883595 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 13:19:42.889918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:42.889881 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:19:42.891839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:42.891816 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:19:42.944014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:42.943978 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rd49z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:19:42.944328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:42.944260 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:19:43.295017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:43.294935 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:45.135685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:45.135646 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:45.825791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:45.825760 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:46.403174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.403039 5515 generic.go:356] "Generic (PLEG): container finished" podID="78162185-1d66-44e4-8652-06a94353866a" containerID="97bd4bb657f18a83420b7d4502e837cb8b7a5a7fedfa7378a5f40cc08a841555" exitCode=0 Oct 30 13:19:46.403174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.403136 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" event={"ID":"78162185-1d66-44e4-8652-06a94353866a","Type":"ContainerDied","Data":"97bd4bb657f18a83420b7d4502e837cb8b7a5a7fedfa7378a5f40cc08a841555"} Oct 30 13:19:46.403704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.403198 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" event={"ID":"78162185-1d66-44e4-8652-06a94353866a","Type":"ContainerDied","Data":"180884a0567d8218c47331a41db1848e7e2453d8f4dd5c156bfca3663a21fda1"} Oct 30 13:19:46.403704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.403220 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="180884a0567d8218c47331a41db1848e7e2453d8f4dd5c156bfca3663a21fda1" Oct 30 13:19:46.404554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.404526 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 13:19:46.502509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502476 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-credential-socket\") pod \"78162185-1d66-44e4-8652-06a94353866a\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " Oct 30 13:19:46.502715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502518 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78162185-1d66-44e4-8652-06a94353866a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78162185-1d66-44e4-8652-06a94353866a-istiod-ca-cert\") pod \"78162185-1d66-44e4-8652-06a94353866a\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " Oct 30 13:19:46.502715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502572 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-kube-api-access-m78h2\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-kube-api-access-m78h2\") pod \"78162185-1d66-44e4-8652-06a94353866a\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " Oct 30 13:19:46.502715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502608 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-istio-token\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-istio-token\") pod \"78162185-1d66-44e4-8652-06a94353866a\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " Oct 30 13:19:46.502715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502651 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-certs\") pod \"78162185-1d66-44e4-8652-06a94353866a\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " Oct 30 13:19:46.502955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502723 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-envoy\") pod \"78162185-1d66-44e4-8652-06a94353866a\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " Oct 30 13:19:46.502955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502783 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78162185-1d66-44e4-8652-06a94353866a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78162185-1d66-44e4-8652-06a94353866a-istio-podinfo\") pod \"78162185-1d66-44e4-8652-06a94353866a\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " Oct 30 13:19:46.502955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502784 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-credential-socket" pod "78162185-1d66-44e4-8652-06a94353866a" (UID: "78162185-1d66-44e4-8652-06a94353866a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:46.502955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502831 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-data\") pod \"78162185-1d66-44e4-8652-06a94353866a\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " Oct 30 13:19:46.502955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.502894 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-socket\") pod \"78162185-1d66-44e4-8652-06a94353866a\" (UID: \"78162185-1d66-44e4-8652-06a94353866a\") " Oct 30 13:19:46.503210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.503015 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78162185-1d66-44e4-8652-06a94353866a-istiod-ca-cert" pod "78162185-1d66-44e4-8652-06a94353866a" (UID: "78162185-1d66-44e4-8652-06a94353866a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:46.503280 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.503240 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-certs" pod "78162185-1d66-44e4-8652-06a94353866a" (UID: "78162185-1d66-44e4-8652-06a94353866a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:46.503335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.503294 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-data" pod "78162185-1d66-44e4-8652-06a94353866a" (UID: "78162185-1d66-44e4-8652-06a94353866a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:46.503502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.503481 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-socket" pod "78162185-1d66-44e4-8652-06a94353866a" (UID: "78162185-1d66-44e4-8652-06a94353866a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:46.503685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.503664 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:46.503748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.503690 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:46.503748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.503707 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78162185-1d66-44e4-8652-06a94353866a-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:46.503748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.503721 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:46.503748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.503734 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:46.505067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.505019 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-envoy" pod "78162185-1d66-44e4-8652-06a94353866a" (UID: "78162185-1d66-44e4-8652-06a94353866a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:46.505234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.505184 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-kube-api-access-m78h2" pod "78162185-1d66-44e4-8652-06a94353866a" (UID: "78162185-1d66-44e4-8652-06a94353866a"). InnerVolumeSpecName "kube-api-access-m78h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:46.505234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.505204 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-istio-token" pod "78162185-1d66-44e4-8652-06a94353866a" (UID: "78162185-1d66-44e4-8652-06a94353866a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:46.505627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.505608 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78162185-1d66-44e4-8652-06a94353866a-istio-podinfo" pod "78162185-1d66-44e4-8652-06a94353866a" (UID: "78162185-1d66-44e4-8652-06a94353866a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:19:46.605035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.605005 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m78h2\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-kube-api-access-m78h2\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:46.605035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.605031 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78162185-1d66-44e4-8652-06a94353866a-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:46.605035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.605041 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78162185-1d66-44e4-8652-06a94353866a-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:46.605315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:46.605049 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78162185-1d66-44e4-8652-06a94353866a-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:19:47.075454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.075424 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:47.330942 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.330868 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:19:47.330942 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.330932 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:19:47.411652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.411620 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z" Oct 30 13:19:47.474686 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.474655 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z"] Oct 30 13:19:47.486938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.486909 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rd49z"] Oct 30 13:19:47.616511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.616425 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6"] Oct 30 13:19:47.617223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.617206 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" Oct 30 13:19:47.617299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.617224 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" Oct 30 13:19:47.617299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.617239 5515 cpu_manager.go:414] "containerMap: removing container" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" Oct 30 13:19:47.617299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.617245 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" Oct 30 13:19:47.617405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.617365 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff42defd-792a-4658-8ac3-e7fef9e4d459" containerName="istio-proxy" Oct 30 13:19:47.617405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.617373 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="78162185-1d66-44e4-8652-06a94353866a" containerName="istio-proxy" Oct 30 13:19:47.621762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.621738 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.639444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.638766 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-182377\"/\"openshift-service-ca.crt\"" Oct 30 13:19:47.639444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.638947 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-182377\"/\"default-dockercfg-khj2z\"" Oct 30 13:19:47.639620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.639163 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-182377\"/\"explicit-model-182377-kserve-self-signed-certs\"" Oct 30 13:19:47.640430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.640195 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-182377\"/\"kube-root-ca.crt\"" Oct 30 13:19:47.642945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.642921 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6"] Oct 30 13:19:47.723132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.723096 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t797s\" (UniqueName: \"kubernetes.io/projected/213fa2bd-c3a3-4ab3-b321-ceae5b268739-kube-api-access-t797s\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.723366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.723163 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-dshm\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.723366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.723319 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-model-cache\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.723510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.723390 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-home\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.723510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.723488 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tmp-dir\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.723614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.723532 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tls-certs\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.797017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.796990 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:47.824863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.824828 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tmp-dir\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.824999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.824900 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tls-certs\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.825061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.825019 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t797s\" (UniqueName: \"kubernetes.io/projected/213fa2bd-c3a3-4ab3-b321-ceae5b268739-kube-api-access-t797s\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.825100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.825057 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tmp-dir\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.825100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.825065 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-dshm\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.825174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.825160 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-model-cache\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.825238 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.825223 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-home\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.825489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.825462 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-model-cache\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.825612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.825538 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-home\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.827295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.827261 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-dshm\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.827518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.827498 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tls-certs\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.849079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.849050 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t797s\" (UniqueName: \"kubernetes.io/projected/213fa2bd-c3a3-4ab3-b321-ceae5b268739-kube-api-access-t797s\") pod \"explicit-model-182377-kserve-78c89b4dc9-79hx6\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:47.935548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:47.935515 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:48.015192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.015149 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78162185-1d66-44e4-8652-06a94353866a" path="/var/lib/kubelet/pods/78162185-1d66-44e4-8652-06a94353866a/volumes" Oct 30 13:19:48.244979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.244946 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6"] Oct 30 13:19:48.246141 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:19:48.246113 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod213fa2bd_c3a3_4ab3_b321_ceae5b268739.slice/crio-dd2cd62718885f3c256655ad70479da41cbb38e164948e47fdb44fc366a3c556 WatchSource:0}: Error finding container dd2cd62718885f3c256655ad70479da41cbb38e164948e47fdb44fc366a3c556: Status 404 returned error can't find the container with id dd2cd62718885f3c256655ad70479da41cbb38e164948e47fdb44fc366a3c556 Oct 30 13:19:48.482541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.482233 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" event={"ID":"213fa2bd-c3a3-4ab3-b321-ceae5b268739","Type":"ContainerStarted","Data":"6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b"} Oct 30 13:19:48.482541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.482301 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" event={"ID":"213fa2bd-c3a3-4ab3-b321-ceae5b268739","Type":"ContainerStarted","Data":"dd2cd62718885f3c256655ad70479da41cbb38e164948e47fdb44fc366a3c556"} Oct 30 13:19:48.525092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.524990 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" podStartSLOduration=1.524976734 podStartE2EDuration="1.524976734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:48.523462709 +0000 UTC m=+2806.650263645" watchObservedRunningTime="2026-10-30 13:19:48.524976734 +0000 UTC m=+2806.651777631" Oct 30 13:19:48.581317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.581285 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk"] Oct 30 13:19:48.584492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.584475 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.602418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.602392 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-1957dc-serving-cert\"" Oct 30 13:19:48.608727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.608698 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk"] Oct 30 13:19:48.638688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.638658 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-metrics-certs\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.638848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.638728 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b564df0-7119-482f-8404-9a78078481aa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.638848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.638818 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfmc\" (UniqueName: \"kubernetes.io/projected/6b564df0-7119-482f-8404-9a78078481aa-kube-api-access-lsfmc\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.638997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.638976 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.739805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.739771 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b564df0-7119-482f-8404-9a78078481aa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.739988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.739822 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfmc\" (UniqueName: \"kubernetes.io/projected/6b564df0-7119-482f-8404-9a78078481aa-kube-api-access-lsfmc\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.739988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.739903 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.739988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.739943 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-metrics-certs\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.740566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.740535 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b564df0-7119-482f-8404-9a78078481aa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.742316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.742295 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.742451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.742354 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-metrics-certs\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.763723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.763697 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfmc\" (UniqueName: \"kubernetes.io/projected/6b564df0-7119-482f-8404-9a78078481aa-kube-api-access-lsfmc\") pod \"maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.773648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.773624 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:48.898749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.898719 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:48.947528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:48.946398 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:49.000251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:49.000219 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:49.222898 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:49.222876 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk"] Oct 30 13:19:49.224714 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:19:49.224687 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b564df0_7119_482f_8404_9a78078481aa.slice/crio-8e091e0831b2dd1d7f9021e360c2c0d94ddb2f7544d46d97040cd9c6e8044b37 WatchSource:0}: Error finding container 8e091e0831b2dd1d7f9021e360c2c0d94ddb2f7544d46d97040cd9c6e8044b37: Status 404 returned error can't find the container with id 8e091e0831b2dd1d7f9021e360c2c0d94ddb2f7544d46d97040cd9c6e8044b37 Oct 30 13:19:49.491068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:49.490980 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" event={"ID":"6b564df0-7119-482f-8404-9a78078481aa","Type":"ContainerStarted","Data":"8e091e0831b2dd1d7f9021e360c2c0d94ddb2f7544d46d97040cd9c6e8044b37"} Oct 30 13:19:50.403758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:50.403718 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:50.500209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:50.500171 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" event={"ID":"6b564df0-7119-482f-8404-9a78078481aa","Type":"ContainerStarted","Data":"57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9"} Oct 30 13:19:50.561964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:50.561914 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" podStartSLOduration=2.275643269 podStartE2EDuration="2.561899825s" totalImagesPullingTime="286.256556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:50.559519826 +0000 UTC m=+2808.686320725" watchObservedRunningTime="2026-10-30 13:19:50.561899825 +0000 UTC m=+2808.688700723" Oct 30 13:19:50.848629 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:50.848546 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:51.507684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:51.507646 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:52.003725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:52.003693 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:19:52.627786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:52.627747 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:52.953591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:52.953549 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:53.424880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.424847 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz"] Oct 30 13:19:53.428738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.428706 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.445713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.445680 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-4db150-openshift-default-dockercfg-rgwxl\"" Oct 30 13:19:53.461280 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.461239 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz"] Oct 30 13:19:53.538810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.538760 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0890391b-13c2-422d-90cf-3ac31659e601-istio-podinfo\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.539038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.538868 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-credential-socket\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.539038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.538919 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-envoy\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.539038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.539027 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-socket\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.539232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.539105 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-istio-token\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.539232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.539137 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9t9h\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-kube-api-access-s9t9h\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.539232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.539177 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-certs\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.539509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.539241 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0890391b-13c2-422d-90cf-3ac31659e601-istiod-ca-cert\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.539509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.539290 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-data\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.607797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.607763 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:53.640047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640014 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0890391b-13c2-422d-90cf-3ac31659e601-istiod-ca-cert\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640059 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-data\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640115 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0890391b-13c2-422d-90cf-3ac31659e601-istio-podinfo\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640197 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-credential-socket\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640246 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-envoy\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640404 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-socket\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640479 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-istio-token\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640510 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9t9h\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-kube-api-access-s9t9h\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640547 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-certs\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640588 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-data\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640772 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-socket\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640870 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0890391b-13c2-422d-90cf-3ac31659e601-istiod-ca-cert\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640882 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-credential-socket\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.640984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.640940 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-certs\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.642678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.642654 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-envoy\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.642786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.642754 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0890391b-13c2-422d-90cf-3ac31659e601-istio-podinfo\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.666217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.666192 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-istio-token\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.666970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.666947 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9t9h\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-kube-api-access-s9t9h\") pod \"e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.747772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.747678 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:53.862286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:53.862214 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:54.077088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:54.077062 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz"] Oct 30 13:19:54.078247 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:19:54.078224 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0890391b_13c2_422d_90cf_3ac31659e601.slice/crio-468313e4863f12f7a50f501bf7b5febb1c177ab67f3b9b7d62fa704a3c640d5c WatchSource:0}: Error finding container 468313e4863f12f7a50f501bf7b5febb1c177ab67f3b9b7d62fa704a3c640d5c: Status 404 returned error can't find the container with id 468313e4863f12f7a50f501bf7b5febb1c177ab67f3b9b7d62fa704a3c640d5c Oct 30 13:19:54.080985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:54.080952 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:54.081054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:54.081017 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:54.081054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:54.081046 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:19:54.452874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:54.452828 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:54.545934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:54.545879 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" event={"ID":"0890391b-13c2-422d-90cf-3ac31659e601","Type":"ContainerStarted","Data":"eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95"} Oct 30 13:19:54.546120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:54.545979 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" event={"ID":"0890391b-13c2-422d-90cf-3ac31659e601","Type":"ContainerStarted","Data":"468313e4863f12f7a50f501bf7b5febb1c177ab67f3b9b7d62fa704a3c640d5c"} Oct 30 13:19:54.621171 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:54.621098 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" podStartSLOduration=1.6210738820000001 podStartE2EDuration="1.621073882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:54.610631552 +0000 UTC m=+2812.737432510" watchObservedRunningTime="2026-10-30 13:19:54.621073882 +0000 UTC m=+2812.747874781" Oct 30 13:19:54.748883 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:54.748700 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:55.755155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:55.755127 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:56.149306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:56.149246 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:56.559652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:56.559557 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:56.560665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:56.560644 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:19:57.245375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:57.245334 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:57.524417 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:57.524345 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:19:57.559507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:57.559474 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:57.936307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:57.936225 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:57.949637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:57.949609 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:58.093254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:58.093225 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:58.583220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:58.583190 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:19:58.990484 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:19:58.990449 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:19:59.047542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:59.047507 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:19:59.346639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:19:59.346558 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:00.140889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:00.140854 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:01.978689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:01.978659 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:03.582592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:03.582562 5515 scope.go:122] "RemoveContainer" containerID="97bd4bb657f18a83420b7d4502e837cb8b7a5a7fedfa7378a5f40cc08a841555" Oct 30 13:20:03.609009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:03.608982 5515 generic.go:356] "Generic (PLEG): container finished" podID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerID="916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61" exitCode=0 Oct 30 13:20:03.609141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:03.609057 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" event={"ID":"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2","Type":"ContainerDied","Data":"916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61"} Oct 30 13:20:03.609611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:03.609593 5515 scope.go:122] "RemoveContainer" containerID="916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61" Oct 30 13:20:03.611655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:03.611626 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:03.611720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:03.611706 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:03.611757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:03.611743 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:03.643580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:03.643547 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:04.625787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:04.625750 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" event={"ID":"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2","Type":"ContainerStarted","Data":"04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4"} Oct 30 13:20:05.136021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:05.135976 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:20:05.136302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:05.136242 5515 patch_prober.go:29] interesting pod/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:20:05.136379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:05.136350 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:20:05.438687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:05.438583 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:06.136688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:06.136650 5515 patch_prober.go:29] interesting pod/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:20:06.137149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:06.136706 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:20:07.104291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:07.100164 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:07.141307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:07.137999 5515 patch_prober.go:29] interesting pod/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:20:07.141307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:07.138062 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:20:07.329005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:07.328970 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:08.136204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:08.136167 5515 patch_prober.go:29] interesting pod/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:20:08.136407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:08.136223 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:20:08.922297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:08.910683 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:09.136196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:09.136160 5515 patch_prober.go:29] interesting pod/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:20:09.136406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:09.136216 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:20:10.136215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:10.136171 5515 patch_prober.go:29] interesting pod/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 13:20:10.136664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:10.136245 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 13:20:10.727997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:10.727957 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:11.064852 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:11.064763 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:11.141627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:11.141595 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:20:11.142121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:11.141947 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:20:11.142799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:11.142779 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:20:11.418682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:11.418649 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6"] Oct 30 13:20:11.577873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:11.577791 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:12.378874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:12.378837 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:12.738826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:12.738790 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:12.989754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:12.989669 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:13.603019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:13.602981 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:14.600396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:14.600359 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:16.286251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:16.286215 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:18.106831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:18.106797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:19.742912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:19.742864 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:21.515793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:21.515756 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:22.270426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:22.270387 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg"] Oct 30 13:20:22.511912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:22.511876 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:22.748815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:22.748768 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:23.457771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:23.457733 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:25.234058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:25.234028 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:26.440043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.439993 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" podUID="213fa2bd-c3a3-4ab3-b321-ceae5b268739" containerName="main" containerID="cri-o://6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b" gracePeriod=45 Oct 30 13:20:26.572631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.572593 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" podUID="213fa2bd-c3a3-4ab3-b321-ceae5b268739" containerName="main" probeResult="failure" output="Get \"https://10.134.0.191:8000/ready\": dial tcp 10.134.0.191:8000: connect: connection refused" Oct 30 13:20:26.802707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.802684 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:20:26.807899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.807873 5515 generic.go:356] "Generic (PLEG): container finished" podID="213fa2bd-c3a3-4ab3-b321-ceae5b268739" containerID="6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b" exitCode=0 Oct 30 13:20:26.808028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.807909 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" Oct 30 13:20:26.808028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.807958 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" event={"ID":"213fa2bd-c3a3-4ab3-b321-ceae5b268739","Type":"ContainerDied","Data":"6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b"} Oct 30 13:20:26.808028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.807998 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6" event={"ID":"213fa2bd-c3a3-4ab3-b321-ceae5b268739","Type":"ContainerDied","Data":"dd2cd62718885f3c256655ad70479da41cbb38e164948e47fdb44fc366a3c556"} Oct 30 13:20:26.808028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.808017 5515 scope.go:122] "RemoveContainer" containerID="6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b" Oct 30 13:20:26.818642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.818624 5515 scope.go:122] "RemoveContainer" containerID="6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b" Oct 30 13:20:26.818934 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:20:26.818912 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b\": container with ID starting with 6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b not found: ID does not exist" containerID="6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b" Oct 30 13:20:26.818985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.818946 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b"} err="failed to get container status \"6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b\": rpc error: code = NotFound desc = could not find container \"6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b\": container with ID starting with 6dc24ce4ca576289301079c967791b89812888a45270d5892dcda6c11f57d97b not found: ID does not exist" Oct 30 13:20:26.912409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.912360 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-home\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-home\") pod \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " Oct 30 13:20:26.912615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.912527 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-dshm\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-dshm\") pod \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " Oct 30 13:20:26.912615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.912561 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tmp-dir\") pod \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " Oct 30 13:20:26.912812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.912647 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213fa2bd-c3a3-4ab3-b321-ceae5b268739-kube-api-access-t797s\" (UniqueName: \"kubernetes.io/projected/213fa2bd-c3a3-4ab3-b321-ceae5b268739-kube-api-access-t797s\") pod \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " Oct 30 13:20:26.912812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.912669 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-home" pod "213fa2bd-c3a3-4ab3-b321-ceae5b268739" (UID: "213fa2bd-c3a3-4ab3-b321-ceae5b268739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:26.912812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.912690 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tls-certs\" (UniqueName: \"kubernetes.io/secret/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tls-certs\") pod \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " Oct 30 13:20:26.912812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.912761 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-model-cache\") pod \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\" (UID: \"213fa2bd-c3a3-4ab3-b321-ceae5b268739\") " Oct 30 13:20:26.913050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.912899 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tmp-dir" pod "213fa2bd-c3a3-4ab3-b321-ceae5b268739" (UID: "213fa2bd-c3a3-4ab3-b321-ceae5b268739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:26.913165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.913132 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-model-cache" pod "213fa2bd-c3a3-4ab3-b321-ceae5b268739" (UID: "213fa2bd-c3a3-4ab3-b321-ceae5b268739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:26.913841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.913800 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:26.913957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.913844 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:26.913957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.913860 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:26.925956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.925911 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213fa2bd-c3a3-4ab3-b321-ceae5b268739-kube-api-access-t797s" pod "213fa2bd-c3a3-4ab3-b321-ceae5b268739" (UID: "213fa2bd-c3a3-4ab3-b321-ceae5b268739"). InnerVolumeSpecName "kube-api-access-t797s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:26.926087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.925974 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-dshm" pod "213fa2bd-c3a3-4ab3-b321-ceae5b268739" (UID: "213fa2bd-c3a3-4ab3-b321-ceae5b268739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:20:26.926144 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:26.926127 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tls-certs" pod "213fa2bd-c3a3-4ab3-b321-ceae5b268739" (UID: "213fa2bd-c3a3-4ab3-b321-ceae5b268739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:27.015329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.015203 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t797s\" (UniqueName: \"kubernetes.io/projected/213fa2bd-c3a3-4ab3-b321-ceae5b268739-kube-api-access-t797s\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:27.015329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.015238 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/213fa2bd-c3a3-4ab3-b321-ceae5b268739-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:27.015329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.015251 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/213fa2bd-c3a3-4ab3-b321-ceae5b268739-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:27.028763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.028727 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:27.155225 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.155192 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6"] Oct 30 13:20:27.166766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.166735 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-182377/explicit-model-182377-kserve-78c89b4dc9-79hx6"] Oct 30 13:20:27.271158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.271066 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" podUID="b9a4b920-aedd-4787-a8fa-876811fa43ed" containerName="payload-pre-processing" containerID="cri-o://aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2" gracePeriod=25 Oct 30 13:20:27.535956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.535898 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:20:27.623234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.623205 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a4b920-aedd-4787-a8fa-876811fa43ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4b920-aedd-4787-a8fa-876811fa43ed-plugins-config-volume\") pod \"b9a4b920-aedd-4787-a8fa-876811fa43ed\" (UID: \"b9a4b920-aedd-4787-a8fa-876811fa43ed\") " Oct 30 13:20:27.623424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.623279 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a4b920-aedd-4787-a8fa-876811fa43ed-kube-api-access-qn9pq\" (UniqueName: \"kubernetes.io/projected/b9a4b920-aedd-4787-a8fa-876811fa43ed-kube-api-access-qn9pq\") pod \"b9a4b920-aedd-4787-a8fa-876811fa43ed\" (UID: \"b9a4b920-aedd-4787-a8fa-876811fa43ed\") " Oct 30 13:20:27.623569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.623548 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a4b920-aedd-4787-a8fa-876811fa43ed-plugins-config-volume" pod "b9a4b920-aedd-4787-a8fa-876811fa43ed" (UID: "b9a4b920-aedd-4787-a8fa-876811fa43ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:27.623928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.623905 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a4b920-aedd-4787-a8fa-876811fa43ed-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:27.625468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.625447 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a4b920-aedd-4787-a8fa-876811fa43ed-kube-api-access-qn9pq" pod "b9a4b920-aedd-4787-a8fa-876811fa43ed" (UID: "b9a4b920-aedd-4787-a8fa-876811fa43ed"). InnerVolumeSpecName "kube-api-access-qn9pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:27.725396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.725368 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn9pq\" (UniqueName: \"kubernetes.io/projected/b9a4b920-aedd-4787-a8fa-876811fa43ed-kube-api-access-qn9pq\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:27.815547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.815478 5515 generic.go:356] "Generic (PLEG): container finished" podID="b9a4b920-aedd-4787-a8fa-876811fa43ed" containerID="aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2" exitCode=0 Oct 30 13:20:27.815547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.815521 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" Oct 30 13:20:27.815856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.815562 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" event={"ID":"b9a4b920-aedd-4787-a8fa-876811fa43ed","Type":"ContainerDied","Data":"aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2"} Oct 30 13:20:27.815856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.815636 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg" event={"ID":"b9a4b920-aedd-4787-a8fa-876811fa43ed","Type":"ContainerDied","Data":"988d764022c3e5405b99c06834749b1a94b3e2c6a157447002a52c16d985a3d7"} Oct 30 13:20:27.815856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.815660 5515 scope.go:122] "RemoveContainer" containerID="aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2" Oct 30 13:20:27.826054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.826032 5515 scope.go:122] "RemoveContainer" containerID="aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2" Oct 30 13:20:27.826404 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:20:27.826379 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2\": container with ID starting with aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2 not found: ID does not exist" containerID="aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2" Oct 30 13:20:27.826478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.826411 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2"} err="failed to get container status \"aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2\": rpc error: code = NotFound desc = could not find container \"aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2\": container with ID starting with aa9e0d819cdaffdc29b2d987ea1152e7e67722f586d6c0fdc2d22bb7b92627d2 not found: ID does not exist" Oct 30 13:20:27.870839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.870807 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg"] Oct 30 13:20:27.874474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:27.874450 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-182377-69d79f5db7-lfjhg"] Oct 30 13:20:28.002962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:28.002923 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213fa2bd-c3a3-4ab3-b321-ceae5b268739" path="/var/lib/kubelet/pods/213fa2bd-c3a3-4ab3-b321-ceae5b268739/volumes" Oct 30 13:20:28.003462 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:28.003447 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a4b920-aedd-4787-a8fa-876811fa43ed" path="/var/lib/kubelet/pods/b9a4b920-aedd-4787-a8fa-876811fa43ed/volumes" Oct 30 13:20:28.386055 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:28.386023 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:20:28.991006 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:20:28.990969 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:20:28.998231 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:28.998198 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:30.841441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:30.841407 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:31.768747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:31.768698 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:32.351083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.351051 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq"] Oct 30 13:20:32.351944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.351919 5515 cpu_manager.go:414] "containerMap: removing container" podUID="213fa2bd-c3a3-4ab3-b321-ceae5b268739" containerName="main" Oct 30 13:20:32.351944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.351941 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213fa2bd-c3a3-4ab3-b321-ceae5b268739" containerName="main" Oct 30 13:20:32.352123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.351955 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b9a4b920-aedd-4787-a8fa-876811fa43ed" containerName="payload-pre-processing" Oct 30 13:20:32.352123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.351961 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a4b920-aedd-4787-a8fa-876811fa43ed" containerName="payload-pre-processing" Oct 30 13:20:32.352123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.352058 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="213fa2bd-c3a3-4ab3-b321-ceae5b268739" containerName="main" Oct 30 13:20:32.352123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.352077 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a4b920-aedd-4787-a8fa-876811fa43ed" containerName="payload-pre-processing" Oct 30 13:20:32.354626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.354610 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.372126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.372096 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-4db150-serving-cert\"" Oct 30 13:20:32.375746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.375718 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq"] Oct 30 13:20:32.497831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.497796 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debfc636-fedd-4960-bc1a-77b6f74e8e88-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.498039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.497935 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.498174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.498126 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knjjg\" (UniqueName: \"kubernetes.io/projected/debfc636-fedd-4960-bc1a-77b6f74e8e88-kube-api-access-knjjg\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.498174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.498169 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-metrics-certs\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.530133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.530098 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:32.607804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.607711 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knjjg\" (UniqueName: \"kubernetes.io/projected/debfc636-fedd-4960-bc1a-77b6f74e8e88-kube-api-access-knjjg\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.607804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.607757 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-metrics-certs\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.608048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.607884 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debfc636-fedd-4960-bc1a-77b6f74e8e88-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.608048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.607963 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.610533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.610487 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debfc636-fedd-4960-bc1a-77b6f74e8e88-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.638573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.633913 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.641108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.639172 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knjjg\" (UniqueName: \"kubernetes.io/projected/debfc636-fedd-4960-bc1a-77b6f74e8e88-kube-api-access-knjjg\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.643317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.643287 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-metrics-certs\") pod \"maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.668810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.668782 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:32.782699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.782662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:32.972298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:32.972238 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq"] Oct 30 13:20:32.973778 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:20:32.973747 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebfc636_fedd_4960_bc1a_77b6f74e8e88.slice/crio-d9a50dd7c518d28ff7a51be9eb5d058166e6fd9f4475091e24b8153bd45a1e59 WatchSource:0}: Error finding container d9a50dd7c518d28ff7a51be9eb5d058166e6fd9f4475091e24b8153bd45a1e59: Status 404 returned error can't find the container with id d9a50dd7c518d28ff7a51be9eb5d058166e6fd9f4475091e24b8153bd45a1e59 Oct 30 13:20:33.198573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:33.198537 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:33.753340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:33.752720 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:33.867798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:33.867761 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" event={"ID":"debfc636-fedd-4960-bc1a-77b6f74e8e88","Type":"ContainerStarted","Data":"a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211"} Oct 30 13:20:33.867971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:33.867817 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" event={"ID":"debfc636-fedd-4960-bc1a-77b6f74e8e88","Type":"ContainerStarted","Data":"d9a50dd7c518d28ff7a51be9eb5d058166e6fd9f4475091e24b8153bd45a1e59"} Oct 30 13:20:33.867971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:33.867854 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:33.915427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:33.915372 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" podStartSLOduration=1.502956926 podStartE2EDuration="1.915358219s" totalImagesPullingTime="412.401293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:33.914258314 +0000 UTC m=+2852.041059215" watchObservedRunningTime="2026-10-30 13:20:33.915358219 +0000 UTC m=+2852.042159117" Oct 30 13:20:34.058225 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:34.058088 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:34.404712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:34.404674 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:36.267937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:36.267904 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:37.998824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:37.998784 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:38.358252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:38.358168 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:39.169877 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.169836 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:39.804573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.804537 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:39.882071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.882028 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:20:39.883703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.883681 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck"] Oct 30 13:20:39.887372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.887352 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:39.909572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.909194 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2d2a0b\"" Oct 30 13:20:39.909837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.909214 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2d2a0b-dockercfg-6d87c\"" Oct 30 13:20:39.913416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.913393 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck"] Oct 30 13:20:39.918326 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.918301 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck\" (UID: \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:39.918510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:39.918485 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wnn\" (UniqueName: \"kubernetes.io/projected/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-kube-api-access-q5wnn\") pod \"payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck\" (UID: \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:40.020666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.020616 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck\" (UID: \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:40.020870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.020791 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wnn\" (UniqueName: \"kubernetes.io/projected/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-kube-api-access-q5wnn\") pod \"payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck\" (UID: \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:40.021860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.021828 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck\" (UID: \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:40.045380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.045342 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wnn\" (UniqueName: \"kubernetes.io/projected/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-kube-api-access-q5wnn\") pod \"payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck\" (UID: \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:40.093770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.093663 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:40.202335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.202299 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:40.346879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.346790 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:40.511460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.511432 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck"] Oct 30 13:20:40.513405 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:20:40.513372 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b1762a_e274_4dad_9f2c_a2c5f6e58380.slice/crio-419f9940f658270c2f32a5b57c8ef026e6123108abde26311a452ff1eae3961e WatchSource:0}: Error finding container 419f9940f658270c2f32a5b57c8ef026e6123108abde26311a452ff1eae3961e: Status 404 returned error can't find the container with id 419f9940f658270c2f32a5b57c8ef026e6123108abde26311a452ff1eae3961e Oct 30 13:20:40.921405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.921366 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" event={"ID":"d8b1762a-e274-4dad-9f2c-a2c5f6e58380","Type":"ContainerStarted","Data":"b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14"} Oct 30 13:20:40.921620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.921431 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:40.921620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.921451 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" event={"ID":"d8b1762a-e274-4dad-9f2c-a2c5f6e58380","Type":"ContainerStarted","Data":"419f9940f658270c2f32a5b57c8ef026e6123108abde26311a452ff1eae3961e"} Oct 30 13:20:40.965696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:40.965639 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" podStartSLOduration=1.965621814 podStartE2EDuration="1.965621814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:40.963425624 +0000 UTC m=+2859.090226525" watchObservedRunningTime="2026-10-30 13:20:40.965621814 +0000 UTC m=+2859.092422738" Oct 30 13:20:42.003642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:42.003610 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:43.514717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:43.514679 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:45.344708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:45.344671 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:46.992936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:46.992897 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:48.717646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:48.717606 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:49.141966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:49.141897 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:49.382368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:49.382326 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:49.614524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:49.614432 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:50.210775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:50.210738 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g"] Oct 30 13:20:50.247339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:50.247308 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz"] Oct 30 13:20:50.414660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:50.414625 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:50.652901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:50.652843 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:50.926348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:50.919069 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:51.230052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.229968 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2"] Oct 30 13:20:51.233786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.233761 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.249176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.249150 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-8740d4-openshift-default-dockercfg-ww9pq\"" Oct 30 13:20:51.265202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.265168 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2"] Oct 30 13:20:51.292772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.292738 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-certs\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.292962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.292797 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-data\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.293006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.292965 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-token\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.293051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.293030 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7b6\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-kube-api-access-vp7b6\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.293107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.293065 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-podinfo\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.293214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.293194 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-socket\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.293315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.293229 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-credential-socket\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.293364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.293332 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-envoy\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.293419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.293388 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istiod-ca-cert\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.394573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.394531 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istiod-ca-cert\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.394573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.394704 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-certs\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.394573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.394772 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-data\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.394573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.394854 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-token\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.394573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.394894 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7b6\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-kube-api-access-vp7b6\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.395367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.394925 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-podinfo\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.395367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.394990 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-socket\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.395367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.395014 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-credential-socket\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.395367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.395064 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-envoy\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.395586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.395489 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istiod-ca-cert\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.395817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.395792 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-socket\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.396925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.396896 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-certs\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.397181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.397158 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-data\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.398291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.398154 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-envoy\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.398823 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.398792 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:51.399110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.399086 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-credential-socket\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.427305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.419850 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-podinfo\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.438905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.437874 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-token\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.438905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.438821 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7b6\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-kube-api-access-vp7b6\") pod \"e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.549776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.549684 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:51.640869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.640834 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:51.884089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.883992 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2"] Oct 30 13:20:51.892329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.891204 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:51.892329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.891344 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:51.892329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.891463 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:20:51.938548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:51.938520 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:20:52.007195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:52.007154 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" event={"ID":"d6f09afe-2478-4f1c-b9c0-dca74dc14765","Type":"ContainerStarted","Data":"886cab76b634ae7f517469edcdafd454cc6cfc848590b059640e420c77b49516"} Oct 30 13:20:52.198355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:52.197937 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:53.024154 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:53.024110 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" event={"ID":"d6f09afe-2478-4f1c-b9c0-dca74dc14765","Type":"ContainerStarted","Data":"a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d"} Oct 30 13:20:53.550034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:53.549996 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:53.556091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:53.556055 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:53.617293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:53.616210 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" podStartSLOduration=2.616189632 podStartE2EDuration="2.616189632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:53.082477342 +0000 UTC m=+2871.209278240" watchObservedRunningTime="2026-10-30 13:20:53.616189632 +0000 UTC m=+2871.742990531" Oct 30 13:20:54.033071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:54.033022 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:54.049818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:54.049785 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:20:54.127891 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:54.127855 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:55.212385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.212335 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" podUID="482d2fa0-cf8d-42d7-b848-d45f375217f2" containerName="payload-processing" containerID="cri-o://9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845" gracePeriod=25 Oct 30 13:20:55.247656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.247617 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" podUID="bb072b8d-a769-4020-ac97-482fe2ee24f4" containerName="payload-pre-processing" containerID="cri-o://3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58" gracePeriod=25 Oct 30 13:20:55.569620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.569583 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x"] Oct 30 13:20:55.570195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.570165 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" podUID="c73f4270-5c4f-4ee4-9164-161143c52466" containerName="maas-api" containerID="cri-o://f5a9ecd33dfe6ffcb6d6ec4c02451fc260c732eefdfcb1a3d890e1c2ba27d3e5" gracePeriod=30 Oct 30 13:20:55.631510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.631473 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:55.729986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.729943 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:20:55.735756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.735729 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:20:55.871290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.867501 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:55.889848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.886861 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482d2fa0-cf8d-42d7-b848-d45f375217f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482d2fa0-cf8d-42d7-b848-d45f375217f2-plugins-config-volume\") pod \"482d2fa0-cf8d-42d7-b848-d45f375217f2\" (UID: \"482d2fa0-cf8d-42d7-b848-d45f375217f2\") " Oct 30 13:20:55.889848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.886969 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb072b8d-a769-4020-ac97-482fe2ee24f4-kube-api-access-rpp82\" (UniqueName: \"kubernetes.io/projected/bb072b8d-a769-4020-ac97-482fe2ee24f4-kube-api-access-rpp82\") pod \"bb072b8d-a769-4020-ac97-482fe2ee24f4\" (UID: \"bb072b8d-a769-4020-ac97-482fe2ee24f4\") " Oct 30 13:20:55.889848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.887022 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb072b8d-a769-4020-ac97-482fe2ee24f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb072b8d-a769-4020-ac97-482fe2ee24f4-plugins-config-volume\") pod \"bb072b8d-a769-4020-ac97-482fe2ee24f4\" (UID: \"bb072b8d-a769-4020-ac97-482fe2ee24f4\") " Oct 30 13:20:55.889848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.887187 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482d2fa0-cf8d-42d7-b848-d45f375217f2-kube-api-access-wxnxf\" (UniqueName: \"kubernetes.io/projected/482d2fa0-cf8d-42d7-b848-d45f375217f2-kube-api-access-wxnxf\") pod \"482d2fa0-cf8d-42d7-b848-d45f375217f2\" (UID: \"482d2fa0-cf8d-42d7-b848-d45f375217f2\") " Oct 30 13:20:55.894293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.891290 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482d2fa0-cf8d-42d7-b848-d45f375217f2-kube-api-access-wxnxf" pod "482d2fa0-cf8d-42d7-b848-d45f375217f2" (UID: "482d2fa0-cf8d-42d7-b848-d45f375217f2"). InnerVolumeSpecName "kube-api-access-wxnxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:55.894293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.891668 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482d2fa0-cf8d-42d7-b848-d45f375217f2-plugins-config-volume" pod "482d2fa0-cf8d-42d7-b848-d45f375217f2" (UID: "482d2fa0-cf8d-42d7-b848-d45f375217f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:55.933484 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.924993 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb072b8d-a769-4020-ac97-482fe2ee24f4-plugins-config-volume" pod "bb072b8d-a769-4020-ac97-482fe2ee24f4" (UID: "bb072b8d-a769-4020-ac97-482fe2ee24f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:55.948293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.939442 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb072b8d-a769-4020-ac97-482fe2ee24f4-kube-api-access-rpp82" pod "bb072b8d-a769-4020-ac97-482fe2ee24f4" (UID: "bb072b8d-a769-4020-ac97-482fe2ee24f4"). InnerVolumeSpecName "kube-api-access-rpp82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:56.050298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.990539 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482d2fa0-cf8d-42d7-b848-d45f375217f2-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:56.050298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.990584 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpp82\" (UniqueName: \"kubernetes.io/projected/bb072b8d-a769-4020-ac97-482fe2ee24f4-kube-api-access-rpp82\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:56.050298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.990601 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb072b8d-a769-4020-ac97-482fe2ee24f4-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:56.050298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:55.990615 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxnxf\" (UniqueName: \"kubernetes.io/projected/482d2fa0-cf8d-42d7-b848-d45f375217f2-kube-api-access-wxnxf\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:56.162876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.162833 5515 generic.go:356] "Generic (PLEG): container finished" podID="482d2fa0-cf8d-42d7-b848-d45f375217f2" containerID="9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845" exitCode=0 Oct 30 13:20:56.163103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.162971 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" event={"ID":"482d2fa0-cf8d-42d7-b848-d45f375217f2","Type":"ContainerDied","Data":"9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845"} Oct 30 13:20:56.163103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.162975 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:20:56.163103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.163021 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" event={"ID":"482d2fa0-cf8d-42d7-b848-d45f375217f2","Type":"ContainerDied","Data":"38408baacd1509bd865be60c3433cf41f75d0c9b2d37af1eb37b5fa9cbaf8050"} Oct 30 13:20:56.163103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.163043 5515 scope.go:122] "RemoveContainer" containerID="9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845" Oct 30 13:20:56.167219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.166190 5515 generic.go:356] "Generic (PLEG): container finished" podID="bb072b8d-a769-4020-ac97-482fe2ee24f4" containerID="3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58" exitCode=0 Oct 30 13:20:56.167219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.166782 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" event={"ID":"bb072b8d-a769-4020-ac97-482fe2ee24f4","Type":"ContainerDied","Data":"3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58"} Oct 30 13:20:56.167219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.166844 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" event={"ID":"bb072b8d-a769-4020-ac97-482fe2ee24f4","Type":"ContainerDied","Data":"1e72ccfafbec8d46d25d2c29630f9407bb8c37a584d9f5a7824fb09aabb41578"} Oct 30 13:20:56.167219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.166928 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz" Oct 30 13:20:56.175091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.174031 5515 generic.go:356] "Generic (PLEG): container finished" podID="c73f4270-5c4f-4ee4-9164-161143c52466" containerID="f5a9ecd33dfe6ffcb6d6ec4c02451fc260c732eefdfcb1a3d890e1c2ba27d3e5" exitCode=0 Oct 30 13:20:56.175091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.174226 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" event={"ID":"c73f4270-5c4f-4ee4-9164-161143c52466","Type":"ContainerDied","Data":"f5a9ecd33dfe6ffcb6d6ec4c02451fc260c732eefdfcb1a3d890e1c2ba27d3e5"} Oct 30 13:20:56.183890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.182156 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:20:56.184123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.183961 5515 scope.go:122] "RemoveContainer" containerID="9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845" Oct 30 13:20:56.184332 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:20:56.184227 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845\": container with ID starting with 9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845 not found: ID does not exist" containerID="9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845" Oct 30 13:20:56.184332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.184260 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845"} err="failed to get container status \"9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845\": rpc error: code = NotFound desc = could not find container \"9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845\": container with ID starting with 9b4b3479af137ff878081476b65e5a16ba3b9e797fb711137ce8472db1f7c845 not found: ID does not exist" Oct 30 13:20:56.184332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.184297 5515 scope.go:122] "RemoveContainer" containerID="3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58" Oct 30 13:20:56.201840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.198885 5515 scope.go:122] "RemoveContainer" containerID="3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58" Oct 30 13:20:56.201840 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:20:56.201758 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58\": container with ID starting with 3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58 not found: ID does not exist" containerID="3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58" Oct 30 13:20:56.201840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.201798 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58"} err="failed to get container status \"3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58\": rpc error: code = NotFound desc = could not find container \"3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58\": container with ID starting with 3198e695d2e10d4edc6a64de44b65b7ff743841d2da21e8948d3b3e67e1aeb58 not found: ID does not exist" Oct 30 13:20:56.238164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.238123 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz"] Oct 30 13:20:56.244567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.244532 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-037a5a-768dd867bc-hw9cz"] Oct 30 13:20:56.324335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.323571 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73f4270-5c4f-4ee4-9164-161143c52466-kube-api-access-c566q\" (UniqueName: \"kubernetes.io/projected/c73f4270-5c4f-4ee4-9164-161143c52466-kube-api-access-c566q\") pod \"c73f4270-5c4f-4ee4-9164-161143c52466\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " Oct 30 13:20:56.324335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.323632 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-metrics-certs\") pod \"c73f4270-5c4f-4ee4-9164-161143c52466\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " Oct 30 13:20:56.324335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.323717 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-maas-api-tls\") pod \"c73f4270-5c4f-4ee4-9164-161143c52466\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " Oct 30 13:20:56.324335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.323787 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73f4270-5c4f-4ee4-9164-161143c52466-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73f4270-5c4f-4ee4-9164-161143c52466-openshift-service-ca\") pod \"c73f4270-5c4f-4ee4-9164-161143c52466\" (UID: \"c73f4270-5c4f-4ee4-9164-161143c52466\") " Oct 30 13:20:56.325190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.325151 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73f4270-5c4f-4ee4-9164-161143c52466-openshift-service-ca" pod "c73f4270-5c4f-4ee4-9164-161143c52466" (UID: "c73f4270-5c4f-4ee4-9164-161143c52466"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:56.327334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.327258 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73f4270-5c4f-4ee4-9164-161143c52466-kube-api-access-c566q" pod "c73f4270-5c4f-4ee4-9164-161143c52466" (UID: "c73f4270-5c4f-4ee4-9164-161143c52466"). InnerVolumeSpecName "kube-api-access-c566q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:56.327461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.327445 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-maas-api-tls" pod "c73f4270-5c4f-4ee4-9164-161143c52466" (UID: "c73f4270-5c4f-4ee4-9164-161143c52466"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:56.328625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.328596 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-metrics-certs" pod "c73f4270-5c4f-4ee4-9164-161143c52466" (UID: "c73f4270-5c4f-4ee4-9164-161143c52466"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:56.345388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.345355 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:56.425886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.425804 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c566q\" (UniqueName: \"kubernetes.io/projected/c73f4270-5c4f-4ee4-9164-161143c52466-kube-api-access-c566q\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:56.425886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.425835 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:56.425886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.425847 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73f4270-5c4f-4ee4-9164-161143c52466-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:56.425886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.425859 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73f4270-5c4f-4ee4-9164-161143c52466-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:20:56.578023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:56.577991 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:57.020293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.020195 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:57.185969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.185943 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" Oct 30 13:20:57.186133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.185970 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x" event={"ID":"c73f4270-5c4f-4ee4-9164-161143c52466","Type":"ContainerDied","Data":"fc70ef8a47ec8ed7eb24edcf20cf790b4160cc2a47f7642c4d6f7e744946c66f"} Oct 30 13:20:57.186133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.186042 5515 scope.go:122] "RemoveContainer" containerID="f5a9ecd33dfe6ffcb6d6ec4c02451fc260c732eefdfcb1a3d890e1c2ba27d3e5" Oct 30 13:20:57.248118 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.248079 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x"] Oct 30 13:20:57.260293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.260230 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-182377-757f6f54-7r28x"] Oct 30 13:20:57.373466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.373373 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579"] Oct 30 13:20:57.374051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.374037 5515 cpu_manager.go:414] "containerMap: removing container" podUID="bb072b8d-a769-4020-ac97-482fe2ee24f4" containerName="payload-pre-processing" Oct 30 13:20:57.374106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.374052 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb072b8d-a769-4020-ac97-482fe2ee24f4" containerName="payload-pre-processing" Oct 30 13:20:57.374106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.374066 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c73f4270-5c4f-4ee4-9164-161143c52466" containerName="maas-api" Oct 30 13:20:57.374106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.374071 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73f4270-5c4f-4ee4-9164-161143c52466" containerName="maas-api" Oct 30 13:20:57.374106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.374089 5515 cpu_manager.go:414] "containerMap: removing container" podUID="482d2fa0-cf8d-42d7-b848-d45f375217f2" containerName="payload-processing" Oct 30 13:20:57.374106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.374094 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482d2fa0-cf8d-42d7-b848-d45f375217f2" containerName="payload-processing" Oct 30 13:20:57.374300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.374223 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb072b8d-a769-4020-ac97-482fe2ee24f4" containerName="payload-pre-processing" Oct 30 13:20:57.374300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.374230 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73f4270-5c4f-4ee4-9164-161143c52466" containerName="maas-api" Oct 30 13:20:57.374300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.374240 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="482d2fa0-cf8d-42d7-b848-d45f375217f2" containerName="payload-processing" Oct 30 13:20:57.377820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.377800 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:20:57.379608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.379564 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:57.423295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.411421 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-8740d4\"" Oct 30 13:20:57.426679 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.426327 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579"] Oct 30 13:20:57.440488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.440457 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4bf\" (UniqueName: \"kubernetes.io/projected/fa629737-3309-43f6-8d6d-bea2f57918de-kube-api-access-lk4bf\") pod \"payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579\" (UID: \"fa629737-3309-43f6-8d6d-bea2f57918de\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:20:57.440719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.440608 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa629737-3309-43f6-8d6d-bea2f57918de-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579\" (UID: \"fa629737-3309-43f6-8d6d-bea2f57918de\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:20:57.446991 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.446962 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5"] Oct 30 13:20:57.450558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.450533 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:20:57.470999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.470970 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-8740d4-dockercfg-fcvt5\"" Oct 30 13:20:57.472505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.472483 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5"] Oct 30 13:20:57.583292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.582865 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa629737-3309-43f6-8d6d-bea2f57918de-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579\" (UID: \"fa629737-3309-43f6-8d6d-bea2f57918de\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:20:57.583292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.583005 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488a7efe-889c-4c6d-a9a1-5fee8a394212-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5\" (UID: \"488a7efe-889c-4c6d-a9a1-5fee8a394212\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:20:57.583292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.583158 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5np\" (UniqueName: \"kubernetes.io/projected/488a7efe-889c-4c6d-a9a1-5fee8a394212-kube-api-access-kf5np\") pod \"payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5\" (UID: \"488a7efe-889c-4c6d-a9a1-5fee8a394212\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:20:57.583292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.583196 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4bf\" (UniqueName: \"kubernetes.io/projected/fa629737-3309-43f6-8d6d-bea2f57918de-kube-api-access-lk4bf\") pod \"payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579\" (UID: \"fa629737-3309-43f6-8d6d-bea2f57918de\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:20:57.607296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.584719 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa629737-3309-43f6-8d6d-bea2f57918de-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579\" (UID: \"fa629737-3309-43f6-8d6d-bea2f57918de\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:20:57.621435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.617234 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:57.642344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.638573 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4bf\" (UniqueName: \"kubernetes.io/projected/fa629737-3309-43f6-8d6d-bea2f57918de-kube-api-access-lk4bf\") pod \"payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579\" (UID: \"fa629737-3309-43f6-8d6d-bea2f57918de\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:20:57.684893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.684858 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488a7efe-889c-4c6d-a9a1-5fee8a394212-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5\" (UID: \"488a7efe-889c-4c6d-a9a1-5fee8a394212\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:20:57.685079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.684977 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5np\" (UniqueName: \"kubernetes.io/projected/488a7efe-889c-4c6d-a9a1-5fee8a394212-kube-api-access-kf5np\") pod \"payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5\" (UID: \"488a7efe-889c-4c6d-a9a1-5fee8a394212\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:20:57.685641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.685612 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488a7efe-889c-4c6d-a9a1-5fee8a394212-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5\" (UID: \"488a7efe-889c-4c6d-a9a1-5fee8a394212\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:20:57.691481 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.691454 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:20:57.735292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.734214 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5np\" (UniqueName: \"kubernetes.io/projected/488a7efe-889c-4c6d-a9a1-5fee8a394212-kube-api-access-kf5np\") pod \"payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5\" (UID: \"488a7efe-889c-4c6d-a9a1-5fee8a394212\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:20:57.764822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:57.764789 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:20:58.003500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.003411 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb072b8d-a769-4020-ac97-482fe2ee24f4" path="/var/lib/kubelet/pods/bb072b8d-a769-4020-ac97-482fe2ee24f4/volumes" Oct 30 13:20:58.004101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.004081 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73f4270-5c4f-4ee4-9164-161143c52466" path="/var/lib/kubelet/pods/c73f4270-5c4f-4ee4-9164-161143c52466/volumes" Oct 30 13:20:58.042015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.041989 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579"] Oct 30 13:20:58.043627 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:20:58.043603 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa629737_3309_43f6_8d6d_bea2f57918de.slice/crio-4d60ff64c5ff65386c26c063fcdc32ce19382e37f19707fae5668de64446dad3 WatchSource:0}: Error finding container 4d60ff64c5ff65386c26c063fcdc32ce19382e37f19707fae5668de64446dad3: Status 404 returned error can't find the container with id 4d60ff64c5ff65386c26c063fcdc32ce19382e37f19707fae5668de64446dad3 Oct 30 13:20:58.108578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.108544 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5"] Oct 30 13:20:58.108982 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:20:58.108957 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488a7efe_889c_4c6d_a9a1_5fee8a394212.slice/crio-dbaa4535f35afd39dd6b1f741e23f937f775681d37a865c686b0cdacd88e40d4 WatchSource:0}: Error finding container dbaa4535f35afd39dd6b1f741e23f937f775681d37a865c686b0cdacd88e40d4: Status 404 returned error can't find the container with id dbaa4535f35afd39dd6b1f741e23f937f775681d37a865c686b0cdacd88e40d4 Oct 30 13:20:58.197026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.196925 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" event={"ID":"488a7efe-889c-4c6d-a9a1-5fee8a394212","Type":"ContainerStarted","Data":"dbaa4535f35afd39dd6b1f741e23f937f775681d37a865c686b0cdacd88e40d4"} Oct 30 13:20:58.198939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.198901 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" event={"ID":"fa629737-3309-43f6-8d6d-bea2f57918de","Type":"ContainerStarted","Data":"34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075"} Oct 30 13:20:58.199061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.198958 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" event={"ID":"fa629737-3309-43f6-8d6d-bea2f57918de","Type":"ContainerStarted","Data":"4d60ff64c5ff65386c26c063fcdc32ce19382e37f19707fae5668de64446dad3"} Oct 30 13:20:58.199061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.199003 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:20:58.241521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.241469 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" podStartSLOduration=1.241454503 podStartE2EDuration="1.241454503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:58.239247449 +0000 UTC m=+2876.366048349" watchObservedRunningTime="2026-10-30 13:20:58.241454503 +0000 UTC m=+2876.368255401" Oct 30 13:20:58.448030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:58.447994 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:20:58.989743 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:20:58.989698 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:20:59.210954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:59.210915 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" event={"ID":"488a7efe-889c-4c6d-a9a1-5fee8a394212","Type":"ContainerStarted","Data":"5ff27a188a9b594686a406dba028f2c3cef939d51c878141133b28d57d3c3767"} Oct 30 13:20:59.211143 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:59.211093 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:20:59.255739 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:20:59.255622 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" podStartSLOduration=2.255602717 podStartE2EDuration="2.255602717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:59.251902131 +0000 UTC m=+2877.378703030" watchObservedRunningTime="2026-10-30 13:20:59.255602717 +0000 UTC m=+2877.382403646" Oct 30 13:21:00.231885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:00.231839 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:01.962783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:01.962711 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:02.231562 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:02.231528 5515 generic.go:356] "Generic (PLEG): container finished" podID="0890391b-13c2-422d-90cf-3ac31659e601" containerID="eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95" exitCode=0 Oct 30 13:21:02.231780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:02.231610 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" event={"ID":"0890391b-13c2-422d-90cf-3ac31659e601","Type":"ContainerDied","Data":"eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95"} Oct 30 13:21:02.232098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:02.232080 5515 scope.go:122] "RemoveContainer" containerID="eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95" Oct 30 13:21:02.233897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:02.233869 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:02.233963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:02.233937 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:02.234002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:02.233968 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:03.240902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:03.240864 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" event={"ID":"0890391b-13c2-422d-90cf-3ac31659e601","Type":"ContainerStarted","Data":"ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266"} Oct 30 13:21:03.643045 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:03.643009 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:03.748638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:03.748606 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:21:03.748935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:03.748904 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 13:21:03.749058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:03.748966 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 13:21:03.940076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:03.940028 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:04.151297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:04.149557 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp"] Oct 30 13:21:04.151297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:04.149821 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" containerID="cri-o://58d877708fa97d374d847e88504e62e3ca7e880e6f819687d5be9ec8d15d44fc" gracePeriod=30 Oct 30 13:21:04.219709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:04.219615 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:04.484522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:04.484439 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:04.748465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:04.748374 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 13:21:04.748465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:04.748440 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 13:21:05.478684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:05.478652 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:05.749127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:05.749038 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 13:21:05.749127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:05.749102 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 13:21:06.290322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.290257 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:06.634232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.634200 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd"] Oct 30 13:21:06.639541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.639516 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.655066 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.655041 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-acf9e2-openshift-default-dockercfg-wcc6s\"" Oct 30 13:21:06.671837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.671809 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd"] Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.747710 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-data\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.747790 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istiod-ca-cert\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.747889 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-kube-api-access-wtxsp\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.747950 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-podinfo\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.748017 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-certs\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.748100 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-socket\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.748105 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.748126 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-credential-socket\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.748151 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.748173 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-envoy\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.748463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.748210 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-token\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.858309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.857237 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-kube-api-access-wtxsp\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.865644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.865615 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-podinfo\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.866451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.866428 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-certs\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.866710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.866692 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-socket\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.866834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.866819 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-credential-socket\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.866983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.866971 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-envoy\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.867149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.867129 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-token\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.867345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.867328 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-data\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.867523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.867510 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istiod-ca-cert\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.868419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.868394 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istiod-ca-cert\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.868549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.859766 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:06.869022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.868999 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-certs\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.869397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.869379 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-socket\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.869786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.869767 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-credential-socket\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.870659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.870371 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-data\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.888534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.888436 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-podinfo\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.890556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.890533 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-envoy\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.891047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.891021 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-kube-api-access-wtxsp\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.903860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.903798 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-token\") pod \"e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:06.961189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:06.961156 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:07.220348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:07.217437 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:07.339522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:07.337516 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd"] Oct 30 13:21:07.339522 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:21:07.338667 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4ca2e_e76e_46b0_b514_a03d9827e132.slice/crio-fea1c9bb0df2ae4b12c648a22f4aa3e985ce40b8190c5ac4a76488c708643f9f WatchSource:0}: Error finding container fea1c9bb0df2ae4b12c648a22f4aa3e985ce40b8190c5ac4a76488c708643f9f: Status 404 returned error can't find the container with id fea1c9bb0df2ae4b12c648a22f4aa3e985ce40b8190c5ac4a76488c708643f9f Oct 30 13:21:07.351857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:07.350219 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:07.351857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:07.350407 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:07.351857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:07.350611 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:21:07.451695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:07.451665 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:07.748551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:07.748511 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 13:21:07.748739 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:07.748566 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 13:21:08.292351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:08.292306 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" event={"ID":"c8b4ca2e-e76e-46b0-b514-a03d9827e132","Type":"ContainerStarted","Data":"2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5"} Oct 30 13:21:08.292868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:08.292373 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" event={"ID":"c8b4ca2e-e76e-46b0-b514-a03d9827e132","Type":"ContainerStarted","Data":"fea1c9bb0df2ae4b12c648a22f4aa3e985ce40b8190c5ac4a76488c708643f9f"} Oct 30 13:21:08.360515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:08.360459 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" podStartSLOduration=2.360443996 podStartE2EDuration="2.360443996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:08.357252645 +0000 UTC m=+2886.484053547" watchObservedRunningTime="2026-10-30 13:21:08.360443996 +0000 UTC m=+2886.487244981" Oct 30 13:21:08.752560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:08.752530 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:21:08.753034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:08.753012 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:21:08.753670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:08.753651 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:21:08.961799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:08.961763 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:08.971892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:08.971851 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:09.183972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.183942 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:09.213924 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.213898 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:21:09.225210 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:21:09.225180 5515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f30d5c_865e_4525_a9dd_b5606c5ea789.slice/crio-58d877708fa97d374d847e88504e62e3ca7e880e6f819687d5be9ec8d15d44fc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f30d5c_865e_4525_a9dd_b5606c5ea789.slice/crio-conmon-58d877708fa97d374d847e88504e62e3ca7e880e6f819687d5be9ec8d15d44fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:21:09.303612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.303576 5515 generic.go:356] "Generic (PLEG): container finished" podID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerID="58d877708fa97d374d847e88504e62e3ca7e880e6f819687d5be9ec8d15d44fc" exitCode=0 Oct 30 13:21:09.304058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.303649 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" event={"ID":"10f30d5c-865e-4525-a9dd-b5606c5ea789","Type":"ContainerDied","Data":"58d877708fa97d374d847e88504e62e3ca7e880e6f819687d5be9ec8d15d44fc"} Oct 30 13:21:09.304058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.303730 5515 scope.go:122] "RemoveContainer" containerID="8036304a9e061c77360fae6bae2527df52f7058164145dbdd3c8ad0ca456f143" Oct 30 13:21:09.304864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.304843 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:09.305959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.305938 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:21:09.454782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.454755 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:21:09.536189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.536154 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-podinfo\") pod \"10f30d5c-865e-4525-a9dd-b5606c5ea789\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " Oct 30 13:21:09.536405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.536200 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-token\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-token\") pod \"10f30d5c-865e-4525-a9dd-b5606c5ea789\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " Oct 30 13:21:09.536405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.536247 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-data\") pod \"10f30d5c-865e-4525-a9dd-b5606c5ea789\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " Oct 30 13:21:09.536405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.536307 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-socket\") pod \"10f30d5c-865e-4525-a9dd-b5606c5ea789\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " Oct 30 13:21:09.536405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.536377 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-kube-api-access-tsr8r\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-kube-api-access-tsr8r\") pod \"10f30d5c-865e-4525-a9dd-b5606c5ea789\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " Oct 30 13:21:09.536843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.536795 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-data" pod "10f30d5c-865e-4525-a9dd-b5606c5ea789" (UID: "10f30d5c-865e-4525-a9dd-b5606c5ea789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:09.536843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.536817 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-socket" pod "10f30d5c-865e-4525-a9dd-b5606c5ea789" (UID: "10f30d5c-865e-4525-a9dd-b5606c5ea789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:09.537097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.536905 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-credential-socket\") pod \"10f30d5c-865e-4525-a9dd-b5606c5ea789\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " Oct 30 13:21:09.537097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.536982 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-envoy\") pod \"10f30d5c-865e-4525-a9dd-b5606c5ea789\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " Oct 30 13:21:09.537221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.537109 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f30d5c-865e-4525-a9dd-b5606c5ea789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f30d5c-865e-4525-a9dd-b5606c5ea789-istiod-ca-cert\") pod \"10f30d5c-865e-4525-a9dd-b5606c5ea789\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " Oct 30 13:21:09.537221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.537205 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-credential-socket" pod "10f30d5c-865e-4525-a9dd-b5606c5ea789" (UID: "10f30d5c-865e-4525-a9dd-b5606c5ea789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:09.537569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.537541 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-certs" pod "10f30d5c-865e-4525-a9dd-b5606c5ea789" (UID: "10f30d5c-865e-4525-a9dd-b5606c5ea789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:09.537766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.537551 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f30d5c-865e-4525-a9dd-b5606c5ea789-istiod-ca-cert" pod "10f30d5c-865e-4525-a9dd-b5606c5ea789" (UID: "10f30d5c-865e-4525-a9dd-b5606c5ea789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:09.537766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.537588 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-certs\") pod \"10f30d5c-865e-4525-a9dd-b5606c5ea789\" (UID: \"10f30d5c-865e-4525-a9dd-b5606c5ea789\") " Oct 30 13:21:09.538478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.538454 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:09.538611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.538483 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:09.538611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.538499 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:09.538611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.538513 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f30d5c-865e-4525-a9dd-b5606c5ea789-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:09.538611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.538527 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:09.540312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.539440 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-envoy" pod "10f30d5c-865e-4525-a9dd-b5606c5ea789" (UID: "10f30d5c-865e-4525-a9dd-b5606c5ea789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:09.540312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.539542 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-podinfo" pod "10f30d5c-865e-4525-a9dd-b5606c5ea789" (UID: "10f30d5c-865e-4525-a9dd-b5606c5ea789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:21:09.540312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.539797 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-token" pod "10f30d5c-865e-4525-a9dd-b5606c5ea789" (UID: "10f30d5c-865e-4525-a9dd-b5606c5ea789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:09.540975 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.540955 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-kube-api-access-tsr8r" pod "10f30d5c-865e-4525-a9dd-b5606c5ea789" (UID: "10f30d5c-865e-4525-a9dd-b5606c5ea789"). InnerVolumeSpecName "kube-api-access-tsr8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:09.639767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.639729 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:09.639767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.639762 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:09.639767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.639772 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsr8r\" (UniqueName: \"kubernetes.io/projected/10f30d5c-865e-4525-a9dd-b5606c5ea789-kube-api-access-tsr8r\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:09.640041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:09.639783 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f30d5c-865e-4525-a9dd-b5606c5ea789-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:10.221289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:10.221237 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:21:10.313217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:10.313133 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" Oct 30 13:21:10.313217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:10.313151 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp" event={"ID":"10f30d5c-865e-4525-a9dd-b5606c5ea789","Type":"ContainerDied","Data":"6bd5d035b1ea75d7f5358e6a5f250319c9c9333955982ef4e729fcaf9e4186a0"} Oct 30 13:21:10.313217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:10.313207 5515 scope.go:122] "RemoveContainer" containerID="58d877708fa97d374d847e88504e62e3ca7e880e6f819687d5be9ec8d15d44fc" Oct 30 13:21:10.367725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:10.367697 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp"] Oct 30 13:21:10.380615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:10.380584 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-182377-openshift-default-5d6bc8dfc7-8lkvp"] Oct 30 13:21:11.163454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:11.163401 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:11.453076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:11.452977 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:12.002585 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:12.002506 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" path="/var/lib/kubelet/pods/10f30d5c-865e-4525-a9dd-b5606c5ea789/volumes" Oct 30 13:21:12.368401 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:12.368251 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:12.929541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:12.929510 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:12.999615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:12.999585 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht"] Oct 30 13:21:13.000336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.000319 5515 cpu_manager.go:414] "containerMap: removing container" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" Oct 30 13:21:13.000400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.000336 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" Oct 30 13:21:13.000400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.000365 5515 cpu_manager.go:414] "containerMap: removing container" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" Oct 30 13:21:13.000400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.000374 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" Oct 30 13:21:13.000504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.000495 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" Oct 30 13:21:13.003809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.003793 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:13.028198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.028168 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht"] Oct 30 13:21:13.033805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.033778 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-acf9e2-dockercfg-8rkkb\"" Oct 30 13:21:13.034168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.033787 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-acf9e2\"" Oct 30 13:21:13.121098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.121061 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht\" (UID: \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:13.121311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.121194 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnxl\" (UniqueName: \"kubernetes.io/projected/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-kube-api-access-zwnxl\") pod \"payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht\" (UID: \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:13.132049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.132014 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62"] Oct 30 13:21:13.133640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.133611 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f30d5c-865e-4525-a9dd-b5606c5ea789" containerName="istio-proxy" Oct 30 13:21:13.137714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.137677 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:13.173476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.173238 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:13.179520 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.178168 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62"] Oct 30 13:21:13.223091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.223011 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b8ed37-561a-4a4f-8ad1-74a065862db4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62\" (UID: \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:13.223249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.223090 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht\" (UID: \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:13.223249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.223182 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pvsd\" (UniqueName: \"kubernetes.io/projected/a3b8ed37-561a-4a4f-8ad1-74a065862db4-kube-api-access-9pvsd\") pod \"payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62\" (UID: \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:13.223249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.223242 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnxl\" (UniqueName: \"kubernetes.io/projected/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-kube-api-access-zwnxl\") pod \"payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht\" (UID: \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:13.223847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.223827 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht\" (UID: \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:13.247279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.247244 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnxl\" (UniqueName: \"kubernetes.io/projected/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-kube-api-access-zwnxl\") pod \"payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht\" (UID: \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:13.318196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.318165 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:13.324862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.324832 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvsd\" (UniqueName: \"kubernetes.io/projected/a3b8ed37-561a-4a4f-8ad1-74a065862db4-kube-api-access-9pvsd\") pod \"payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62\" (UID: \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:13.324998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.324986 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b8ed37-561a-4a4f-8ad1-74a065862db4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62\" (UID: \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:13.325665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.325647 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b8ed37-561a-4a4f-8ad1-74a065862db4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62\" (UID: \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:13.348717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.348683 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvsd\" (UniqueName: \"kubernetes.io/projected/a3b8ed37-561a-4a4f-8ad1-74a065862db4-kube-api-access-9pvsd\") pod \"payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62\" (UID: \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:13.475501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.475422 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:13.657141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.657113 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht"] Oct 30 13:21:13.657821 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:21:13.657797 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfe827c_ab2e_4d55_9b10_ad2e83dd742e.slice/crio-0972458c49615e1144090b73e8c3bf22987020880222f1a416a7df03e7cef19f WatchSource:0}: Error finding container 0972458c49615e1144090b73e8c3bf22987020880222f1a416a7df03e7cef19f: Status 404 returned error can't find the container with id 0972458c49615e1144090b73e8c3bf22987020880222f1a416a7df03e7cef19f Oct 30 13:21:13.791911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:13.791887 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62"] Oct 30 13:21:13.793346 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:21:13.793321 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b8ed37_561a_4a4f_8ad1_74a065862db4.slice/crio-c247959e582c001d6dc3653bf50ed116090b7c193d46f2c79a96459be714f48a WatchSource:0}: Error finding container c247959e582c001d6dc3653bf50ed116090b7c193d46f2c79a96459be714f48a: Status 404 returned error can't find the container with id c247959e582c001d6dc3653bf50ed116090b7c193d46f2c79a96459be714f48a Oct 30 13:21:14.005627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.005554 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:14.345486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.345399 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" event={"ID":"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e","Type":"ContainerStarted","Data":"b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33"} Oct 30 13:21:14.345486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.345467 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" event={"ID":"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e","Type":"ContainerStarted","Data":"0972458c49615e1144090b73e8c3bf22987020880222f1a416a7df03e7cef19f"} Oct 30 13:21:14.345726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.345633 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:14.347230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.347202 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" event={"ID":"a3b8ed37-561a-4a4f-8ad1-74a065862db4","Type":"ContainerStarted","Data":"d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693"} Oct 30 13:21:14.347397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.347255 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:14.347397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.347294 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" event={"ID":"a3b8ed37-561a-4a4f-8ad1-74a065862db4","Type":"ContainerStarted","Data":"c247959e582c001d6dc3653bf50ed116090b7c193d46f2c79a96459be714f48a"} Oct 30 13:21:14.390902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.390840 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" podStartSLOduration=2.39082238 podStartE2EDuration="2.39082238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:14.389077638 +0000 UTC m=+2892.515878536" watchObservedRunningTime="2026-10-30 13:21:14.39082238 +0000 UTC m=+2892.517623279" Oct 30 13:21:14.437574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.437516 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" podStartSLOduration=1.437501178 podStartE2EDuration="1.437501178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:14.436409422 +0000 UTC m=+2892.563210320" watchObservedRunningTime="2026-10-30 13:21:14.437501178 +0000 UTC m=+2892.564302075" Oct 30 13:21:14.582153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.582117 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s"] Oct 30 13:21:14.586235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.586212 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.604311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.604231 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2d2a0b-serving-cert\"" Oct 30 13:21:14.611961 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.611932 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s"] Oct 30 13:21:14.747386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.747350 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.747386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.747391 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78grm\" (UniqueName: \"kubernetes.io/projected/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-kube-api-access-78grm\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.747651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.747418 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.747651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.747564 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-metrics-certs\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.804671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.804632 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:14.848871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.848839 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78grm\" (UniqueName: \"kubernetes.io/projected/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-kube-api-access-78grm\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.849051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.848880 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.849051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.848933 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-metrics-certs\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.849211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.849119 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.849784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.849758 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.851608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.851584 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-metrics-certs\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.851775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.851754 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.872759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.872696 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78grm\" (UniqueName: \"kubernetes.io/projected/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-kube-api-access-78grm\") pod \"maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:14.902521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:14.902488 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:15.088581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:15.088549 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:15.206169 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:15.206135 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s"] Oct 30 13:21:15.207346 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:21:15.207319 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff901dc4_d932_4a95_8e36_e1ccc2ab4050.slice/crio-56044dd85482fe0258474fcad44ef0fbeb80a21fc7ad450f097717a088c641ee WatchSource:0}: Error finding container 56044dd85482fe0258474fcad44ef0fbeb80a21fc7ad450f097717a088c641ee: Status 404 returned error can't find the container with id 56044dd85482fe0258474fcad44ef0fbeb80a21fc7ad450f097717a088c641ee Oct 30 13:21:15.364939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:15.364904 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" event={"ID":"ff901dc4-d932-4a95-8e36-e1ccc2ab4050","Type":"ContainerStarted","Data":"56044dd85482fe0258474fcad44ef0fbeb80a21fc7ad450f097717a088c641ee"} Oct 30 13:21:15.478215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:15.478132 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:16.084290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:16.080725 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:16.373934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:16.373840 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" event={"ID":"ff901dc4-d932-4a95-8e36-e1ccc2ab4050","Type":"ContainerStarted","Data":"d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d"} Oct 30 13:21:16.374355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:16.373936 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:16.377582 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:16.377557 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:16.419987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:16.419930 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" podStartSLOduration=2.122671155 podStartE2EDuration="2.419915409s" totalImagesPullingTime="297.244254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:16.416237789 +0000 UTC m=+2894.543038690" watchObservedRunningTime="2026-10-30 13:21:16.419915409 +0000 UTC m=+2894.546716307" Oct 30 13:21:16.665262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:16.665234 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:18.420050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:18.420017 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:20.277518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:20.277484 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:20.515589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:20.515551 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:21.915438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:21.915400 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:21:22.010772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.010733 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:22.239342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.234922 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:22.252373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.252335 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8"] Oct 30 13:21:22.284086 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.284045 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8"] Oct 30 13:21:22.284443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.284409 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:22.304518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.304486 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-75d385\"" Oct 30 13:21:22.304862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.304840 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-75d385-dockercfg-kbc9x\"" Oct 30 13:21:22.380426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.380392 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4d8n\" (UniqueName: \"kubernetes.io/projected/60b7cce7-5160-459f-b487-a0c3074697cd-kube-api-access-v4d8n\") pod \"payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8\" (UID: \"60b7cce7-5160-459f-b487-a0c3074697cd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:22.380598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.380527 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b7cce7-5160-459f-b487-a0c3074697cd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8\" (UID: \"60b7cce7-5160-459f-b487-a0c3074697cd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:22.387112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.387085 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:21:22.483919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.481493 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4d8n\" (UniqueName: \"kubernetes.io/projected/60b7cce7-5160-459f-b487-a0c3074697cd-kube-api-access-v4d8n\") pod \"payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8\" (UID: \"60b7cce7-5160-459f-b487-a0c3074697cd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:22.483919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.481608 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b7cce7-5160-459f-b487-a0c3074697cd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8\" (UID: \"60b7cce7-5160-459f-b487-a0c3074697cd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:22.483919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.483178 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b7cce7-5160-459f-b487-a0c3074697cd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8\" (UID: \"60b7cce7-5160-459f-b487-a0c3074697cd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:22.524335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.524239 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4d8n\" (UniqueName: \"kubernetes.io/projected/60b7cce7-5160-459f-b487-a0c3074697cd-kube-api-access-v4d8n\") pod \"payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8\" (UID: \"60b7cce7-5160-459f-b487-a0c3074697cd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:22.611203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.611172 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:22.878366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.878249 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:22.923619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:22.923594 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8"] Oct 30 13:21:22.927356 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:21:22.927325 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b7cce7_5160_459f_b487_a0c3074697cd.slice/crio-871c9a9df2a2e0ab8e99ac5369c7005ff5ba3486461343bec2ddd1f96e190c02 WatchSource:0}: Error finding container 871c9a9df2a2e0ab8e99ac5369c7005ff5ba3486461343bec2ddd1f96e190c02: Status 404 returned error can't find the container with id 871c9a9df2a2e0ab8e99ac5369c7005ff5ba3486461343bec2ddd1f96e190c02 Oct 30 13:21:23.093794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:23.093763 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:23.510240 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:23.510198 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" event={"ID":"60b7cce7-5160-459f-b487-a0c3074697cd","Type":"ContainerStarted","Data":"e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1"} Oct 30 13:21:23.510485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:23.510300 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" event={"ID":"60b7cce7-5160-459f-b487-a0c3074697cd","Type":"ContainerStarted","Data":"871c9a9df2a2e0ab8e99ac5369c7005ff5ba3486461343bec2ddd1f96e190c02"} Oct 30 13:21:23.511430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:23.510696 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:23.557632 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:23.557583 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" podStartSLOduration=1.557569188 podStartE2EDuration="1.557569188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:23.555173522 +0000 UTC m=+2901.681974419" watchObservedRunningTime="2026-10-30 13:21:23.557569188 +0000 UTC m=+2901.684370086" Oct 30 13:21:23.560749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:23.560720 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:24.963226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:24.963193 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:25.368512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:25.368429 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:21:25.368512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:25.368509 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:21:26.175629 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:26.175591 5515 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod482d2fa0-cf8d-42d7-b848-d45f375217f2"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod482d2fa0-cf8d-42d7-b848-d45f375217f2] : Timed out while waiting for systemd to remove kubepods-burstable-pod482d2fa0_cf8d_42d7_b848_d45f375217f2.slice" Oct 30 13:21:26.175629 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:21:26.175630 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod482d2fa0-cf8d-42d7-b848-d45f375217f2] : unable to destroy cgroup paths for cgroup [kubepods burstable pod482d2fa0-cf8d-42d7-b848-d45f375217f2] : Timed out while waiting for systemd to remove kubepods-burstable-pod482d2fa0_cf8d_42d7_b848_d45f375217f2.slice" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" podUID="482d2fa0-cf8d-42d7-b848-d45f375217f2" Oct 30 13:21:26.531730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:26.531597 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g" Oct 30 13:21:26.611017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:26.610976 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g"] Oct 30 13:21:26.614916 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:26.614891 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-037a5a-6596f65f9d-mrw4g"] Oct 30 13:21:26.726101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:26.726061 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:28.003262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:28.003229 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482d2fa0-cf8d-42d7-b848-d45f375217f2" path="/var/lib/kubelet/pods/482d2fa0-cf8d-42d7-b848-d45f375217f2/volumes" Oct 30 13:21:28.658431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:28.658397 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:28.989924 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:21:28.989846 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:21:30.299771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:30.299741 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:30.760818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:30.760775 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:30.988528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:30.988494 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:30.990016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:30.989983 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk"] Oct 30 13:21:30.990284 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:30.990235 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" containerID="cri-o://04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4" gracePeriod=30 Oct 30 13:21:31.204670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:31.204639 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:31.999784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:31.999752 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:33.065751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:33.065712 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:33.626747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:33.626713 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:33.888770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:33.888730 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:34.094112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:34.094075 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:34.318813 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:34.318742 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:34.521645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:34.521612 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:21:35.814553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:35.814523 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:36.262780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.262757 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:21:36.369435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369359 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-socket\") pod \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " Oct 30 13:21:36.369435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369410 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-certs\") pod \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " Oct 30 13:21:36.369692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369464 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-data\") pod \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " Oct 30 13:21:36.369692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369491 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istiod-ca-cert\") pod \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " Oct 30 13:21:36.369692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369548 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-kube-api-access-75jbg\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-kube-api-access-75jbg\") pod \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " Oct 30 13:21:36.369692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369628 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-socket" pod "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" (UID: "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:36.369692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369672 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-podinfo\") pod \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " Oct 30 13:21:36.369971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369716 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-credential-socket\") pod \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " Oct 30 13:21:36.369971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369721 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-certs" pod "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" (UID: "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:36.369971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369739 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-data" pod "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" (UID: "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:36.369971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369749 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-token\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-token\") pod \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " Oct 30 13:21:36.369971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369818 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-envoy\") pod \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\" (UID: \"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2\") " Oct 30 13:21:36.369971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369929 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-credential-socket" pod "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" (UID: "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:36.370314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.369980 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istiod-ca-cert" pod "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" (UID: "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:36.370444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.370421 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:36.370622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.370600 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:36.370713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.370626 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:36.370713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.370640 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:36.370713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.370655 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:36.371909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.371888 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-token" pod "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" (UID: "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:36.371989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.371932 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-kube-api-access-75jbg" pod "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" (UID: "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2"). InnerVolumeSpecName "kube-api-access-75jbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:36.371989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.371947 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-envoy" pod "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" (UID: "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:36.371989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.371957 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-podinfo" pod "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" (UID: "9fe1827e-d5b2-4052-a34f-7552b7a6b4b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:21:36.471903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.471868 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:36.471903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.471901 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:36.472155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.471924 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:36.472155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.471939 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75jbg\" (UniqueName: \"kubernetes.io/projected/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2-kube-api-access-75jbg\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:21:36.654905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.654873 5515 generic.go:356] "Generic (PLEG): container finished" podID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerID="04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4" exitCode=0 Oct 30 13:21:36.655084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.654927 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" Oct 30 13:21:36.655084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.654957 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" event={"ID":"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2","Type":"ContainerDied","Data":"04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4"} Oct 30 13:21:36.655084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.655012 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk" event={"ID":"9fe1827e-d5b2-4052-a34f-7552b7a6b4b2","Type":"ContainerDied","Data":"e383b8fa9beaea3c2dfa89ee451ff4a1040ad7cd0f15d97b2915ebcd666368ee"} Oct 30 13:21:36.655084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.655037 5515 scope.go:122] "RemoveContainer" containerID="04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4" Oct 30 13:21:36.666374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.666353 5515 scope.go:122] "RemoveContainer" containerID="916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61" Oct 30 13:21:36.676597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.676576 5515 scope.go:122] "RemoveContainer" containerID="04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4" Oct 30 13:21:36.676879 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:21:36.676861 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4\": container with ID starting with 04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4 not found: ID does not exist" containerID="04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4" Oct 30 13:21:36.676925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.676888 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4"} err="failed to get container status \"04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4\": rpc error: code = NotFound desc = could not find container \"04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4\": container with ID starting with 04b238688bd296de762487c4e854c28367ded27e792534e1dcef34ed9f6c81d4 not found: ID does not exist" Oct 30 13:21:36.676925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.676904 5515 scope.go:122] "RemoveContainer" containerID="916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61" Oct 30 13:21:36.677134 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:21:36.677118 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61\": container with ID starting with 916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61 not found: ID does not exist" containerID="916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61" Oct 30 13:21:36.677193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.677137 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61"} err="failed to get container status \"916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61\": rpc error: code = NotFound desc = could not find container \"916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61\": container with ID starting with 916d31d8a7ae75415b1da8115e6332c4c556904d21d4445d5a0376e0616b2c61 not found: ID does not exist" Oct 30 13:21:36.709586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.709555 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk"] Oct 30 13:21:36.721829 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:36.721797 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-037a5a-openshift-default-6d5b9745cc-6wkwk"] Oct 30 13:21:37.060128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:37.060024 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:37.311223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:37.311142 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:37.695059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:37.695010 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:38.004022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.003935 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" path="/var/lib/kubelet/pods/9fe1827e-d5b2-4052-a34f-7552b7a6b4b2/volumes" Oct 30 13:21:38.758879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.758843 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:38.824162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.824123 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz"] Oct 30 13:21:38.824896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.824881 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" Oct 30 13:21:38.824947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.824897 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" Oct 30 13:21:38.824947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.824922 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" Oct 30 13:21:38.824947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.824928 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" Oct 30 13:21:38.825059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.825048 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" Oct 30 13:21:38.828719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.828700 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:38.861957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.861924 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0505b0-dockercfg-shbxx\"" Oct 30 13:21:38.862163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.861989 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0505b0\"" Oct 30 13:21:38.883373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.883333 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz"] Oct 30 13:21:38.908144 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.908107 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee9551b-f70c-48cd-8b53-76883f77cb62-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz\" (UID: \"0ee9551b-f70c-48cd-8b53-76883f77cb62\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:38.908380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.908278 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jh2b\" (UniqueName: \"kubernetes.io/projected/0ee9551b-f70c-48cd-8b53-76883f77cb62-kube-api-access-9jh2b\") pod \"payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz\" (UID: \"0ee9551b-f70c-48cd-8b53-76883f77cb62\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:38.945300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.943882 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx"] Oct 30 13:21:38.945300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.945131 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe1827e-d5b2-4052-a34f-7552b7a6b4b2" containerName="istio-proxy" Oct 30 13:21:38.963290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.948416 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:38.970771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.970738 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:38.976889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:38.976858 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx"] Oct 30 13:21:39.013196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.012811 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9dkd\" (UniqueName: \"kubernetes.io/projected/d841893d-a574-4fb4-bfed-b26bf84074d9-kube-api-access-j9dkd\") pod \"payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx\" (UID: \"d841893d-a574-4fb4-bfed-b26bf84074d9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:39.013569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.013458 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jh2b\" (UniqueName: \"kubernetes.io/projected/0ee9551b-f70c-48cd-8b53-76883f77cb62-kube-api-access-9jh2b\") pod \"payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz\" (UID: \"0ee9551b-f70c-48cd-8b53-76883f77cb62\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:39.013569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.013504 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841893d-a574-4fb4-bfed-b26bf84074d9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx\" (UID: \"d841893d-a574-4fb4-bfed-b26bf84074d9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:39.013794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.013678 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee9551b-f70c-48cd-8b53-76883f77cb62-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz\" (UID: \"0ee9551b-f70c-48cd-8b53-76883f77cb62\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:39.014842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.014742 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee9551b-f70c-48cd-8b53-76883f77cb62-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz\" (UID: \"0ee9551b-f70c-48cd-8b53-76883f77cb62\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:39.069845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.069815 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jh2b\" (UniqueName: \"kubernetes.io/projected/0ee9551b-f70c-48cd-8b53-76883f77cb62-kube-api-access-9jh2b\") pod \"payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz\" (UID: \"0ee9551b-f70c-48cd-8b53-76883f77cb62\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:39.115178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.115141 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9dkd\" (UniqueName: \"kubernetes.io/projected/d841893d-a574-4fb4-bfed-b26bf84074d9-kube-api-access-j9dkd\") pod \"payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx\" (UID: \"d841893d-a574-4fb4-bfed-b26bf84074d9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:39.115407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.115210 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841893d-a574-4fb4-bfed-b26bf84074d9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx\" (UID: \"d841893d-a574-4fb4-bfed-b26bf84074d9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:39.115993 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.115969 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841893d-a574-4fb4-bfed-b26bf84074d9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx\" (UID: \"d841893d-a574-4fb4-bfed-b26bf84074d9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:39.142786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.142755 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9dkd\" (UniqueName: \"kubernetes.io/projected/d841893d-a574-4fb4-bfed-b26bf84074d9-kube-api-access-j9dkd\") pod \"payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx\" (UID: \"d841893d-a574-4fb4-bfed-b26bf84074d9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:39.144639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.144614 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:39.316998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.316915 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:39.468294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.453826 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:39.500037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.499661 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz"] Oct 30 13:21:39.503444 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:21:39.501305 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee9551b_f70c_48cd_8b53_76883f77cb62.slice/crio-3ff3437012b75aa02760b948e187e110791a194581105782cdbdcabcd98a2dbc WatchSource:0}: Error finding container 3ff3437012b75aa02760b948e187e110791a194581105782cdbdcabcd98a2dbc: Status 404 returned error can't find the container with id 3ff3437012b75aa02760b948e187e110791a194581105782cdbdcabcd98a2dbc Oct 30 13:21:39.641098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.641068 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx"] Oct 30 13:21:39.641905 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:21:39.641880 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd841893d_a574_4fb4_bfed_b26bf84074d9.slice/crio-6f7e8192bd79fd8d6f68cbb701f99e798b0ca68d1fe281c870c3f8a095c83387 WatchSource:0}: Error finding container 6f7e8192bd79fd8d6f68cbb701f99e798b0ca68d1fe281c870c3f8a095c83387: Status 404 returned error can't find the container with id 6f7e8192bd79fd8d6f68cbb701f99e798b0ca68d1fe281c870c3f8a095c83387 Oct 30 13:21:39.695189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.695154 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" event={"ID":"d841893d-a574-4fb4-bfed-b26bf84074d9","Type":"ContainerStarted","Data":"6f7e8192bd79fd8d6f68cbb701f99e798b0ca68d1fe281c870c3f8a095c83387"} Oct 30 13:21:39.696912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.696882 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" event={"ID":"0ee9551b-f70c-48cd-8b53-76883f77cb62","Type":"ContainerStarted","Data":"74384314be4506088906ff3a054c81afc9ba835042f5109b98bb87b6040a9fdd"} Oct 30 13:21:39.697039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.696932 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" event={"ID":"0ee9551b-f70c-48cd-8b53-76883f77cb62","Type":"ContainerStarted","Data":"3ff3437012b75aa02760b948e187e110791a194581105782cdbdcabcd98a2dbc"} Oct 30 13:21:39.697039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.696997 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:39.739880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:39.739835 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" podStartSLOduration=1.7398196910000001 podStartE2EDuration="1.739819691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:39.73805791 +0000 UTC m=+2917.864858808" watchObservedRunningTime="2026-10-30 13:21:39.739819691 +0000 UTC m=+2917.866620589" Oct 30 13:21:40.706309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:40.706262 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" event={"ID":"d841893d-a574-4fb4-bfed-b26bf84074d9","Type":"ContainerStarted","Data":"e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7"} Oct 30 13:21:40.706822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:40.706350 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:40.748682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:40.748632 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" podStartSLOduration=2.748618761 podStartE2EDuration="2.748618761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:40.747851631 +0000 UTC m=+2918.874652529" watchObservedRunningTime="2026-10-30 13:21:40.748618761 +0000 UTC m=+2918.875419658" Oct 30 13:21:41.151323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:41.151289 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:42.748558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:42.748512 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:44.509374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:44.509341 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:46.224709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:46.224676 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:47.899796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:47.899764 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:49.632128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:49.632095 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:49.888899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:49.888810 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:50.721449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:50.721419 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:21:51.333612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:51.333578 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:51.717656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:51.717622 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:21:53.612644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:53.612608 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:54.700802 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:54.700759 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:56.409765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:56.409726 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:58.249373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:58.249340 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:58.419108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:58.419069 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:21:59.001089 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:21:59.001051 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:21:59.762867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.762825 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:21:59.855283 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.855236 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c"] Oct 30 13:21:59.860743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.860720 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:21:59.890625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.890598 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-75d385-serving-cert\"" Oct 30 13:21:59.925334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.921993 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c"] Oct 30 13:21:59.926302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.926151 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:21:59.926458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.926410 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-metrics-certs\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:21:59.926529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.926459 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-maas-api-tls\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:21:59.926595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.926565 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj96p\" (UniqueName: \"kubernetes.io/projected/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-kube-api-access-mj96p\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:21:59.992220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:21:59.992181 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:00.028019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.027942 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:00.028019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.027982 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-metrics-certs\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:00.028019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.028017 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-maas-api-tls\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:00.028376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.028231 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj96p\" (UniqueName: \"kubernetes.io/projected/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-kube-api-access-mj96p\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:00.028805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.028776 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:00.031145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.031124 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-maas-api-tls\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:00.031296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.031244 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-metrics-certs\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:00.052169 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.052145 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj96p\" (UniqueName: \"kubernetes.io/projected/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-kube-api-access-mj96p\") pod \"maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:00.175701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.175664 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:00.213223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.213129 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:00.536248 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:22:00.536197 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bacc714_649c_4d1b_9ee5_fce729d9d7ef.slice/crio-4616f5dafbd38da8a587ee1b0ef8d0bf47bc583ff6bc61a255e38f938aee3763 WatchSource:0}: Error finding container 4616f5dafbd38da8a587ee1b0ef8d0bf47bc583ff6bc61a255e38f938aee3763: Status 404 returned error can't find the container with id 4616f5dafbd38da8a587ee1b0ef8d0bf47bc583ff6bc61a255e38f938aee3763 Oct 30 13:22:00.539129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.539102 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c"] Oct 30 13:22:00.872470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:00.872382 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" event={"ID":"0bacc714-649c-4d1b-9ee5-fce729d9d7ef","Type":"ContainerStarted","Data":"4616f5dafbd38da8a587ee1b0ef8d0bf47bc583ff6bc61a255e38f938aee3763"} Oct 30 13:22:01.053292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:01.044736 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:01.262303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:01.262254 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:01.524902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:01.524816 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:01.751745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:01.751709 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:01.950300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:01.940990 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" event={"ID":"0bacc714-649c-4d1b-9ee5-fce729d9d7ef","Type":"ContainerStarted","Data":"8625d9fbe1d1365798ce95063671f81fec0c5190e78205c10c375b4752cec5a1"} Oct 30 13:22:01.960312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:01.958690 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:01.972930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:01.972857 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:02.016291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:02.016210 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" podStartSLOduration=2.575241566 podStartE2EDuration="3.016192741s" totalImagesPullingTime="440.951175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:02.014086523 +0000 UTC m=+2940.140887424" watchObservedRunningTime="2026-10-30 13:22:02.016192741 +0000 UTC m=+2940.142993658" Oct 30 13:22:02.535186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:02.535150 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:02.749083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:02.749046 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:03.964642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:03.964567 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:04.182784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.182753 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:04.284923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.284845 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl"] Oct 30 13:22:04.289162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.289145 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.313923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.313892 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"kube-root-ca.crt\"" Oct 30 13:22:04.314944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.314393 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d2a0b\"" Oct 30 13:22:04.315179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.314449 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"openshift-service-ca.crt\"" Oct 30 13:22:04.315304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.314561 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"routing-overlay\"" Oct 30 13:22:04.322714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.322202 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl"] Oct 30 13:22:04.381398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.381363 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-f0f7bfc91557\"" Oct 30 13:22:04.381623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.381582 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-e2e--e20aaa17-dockercfg-2m475\"" Oct 30 13:22:04.478292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.469728 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knl9s\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-kube-api-access-knl9s\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.478292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.469782 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.478292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.469826 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.478292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.469971 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.571761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.571674 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.571938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.571799 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knl9s\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-kube-api-access-knl9s\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.571938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.571827 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.571938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.571864 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.572422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.572395 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.572822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.572800 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.574519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.574494 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.616291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.608966 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knl9s\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-kube-api-access-knl9s\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.628857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.628827 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:04.939323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.939298 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl"] Oct 30 13:22:04.941791 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:22:04.941766 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbb0f15_db5c_4488_83f7_62d6347c1dd0.slice/crio-f2d330b841727648f873a65a53d949e7d6f34132ae6f42ff6b9fb0714d99e2ec WatchSource:0}: Error finding container f2d330b841727648f873a65a53d949e7d6f34132ae6f42ff6b9fb0714d99e2ec: Status 404 returned error can't find the container with id f2d330b841727648f873a65a53d949e7d6f34132ae6f42ff6b9fb0714d99e2ec Oct 30 13:22:04.970046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:04.970003 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" event={"ID":"9cbb0f15-db5c-4488-83f7-62d6347c1dd0","Type":"ContainerStarted","Data":"f2d330b841727648f873a65a53d949e7d6f34132ae6f42ff6b9fb0714d99e2ec"} Oct 30 13:22:05.680761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:05.680726 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:05.980925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:05.980826 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" event={"ID":"9cbb0f15-db5c-4488-83f7-62d6347c1dd0","Type":"ContainerStarted","Data":"2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb"} Oct 30 13:22:05.980925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:05.980905 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:06.026932 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:06.026878 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" podStartSLOduration=2.026863391 podStartE2EDuration="2.026863391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:06.024241212 +0000 UTC m=+2944.151042110" watchObservedRunningTime="2026-10-30 13:22:06.026863391 +0000 UTC m=+2944.153664297" Oct 30 13:22:07.368140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:07.368078 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:07.955258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:07.955230 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:22:09.017414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:09.017377 5515 generic.go:356] "Generic (PLEG): container finished" podID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerID="a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d" exitCode=0 Oct 30 13:22:09.018352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:09.017693 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" event={"ID":"d6f09afe-2478-4f1c-b9c0-dca74dc14765","Type":"ContainerDied","Data":"a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d"} Oct 30 13:22:09.019067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:09.019049 5515 scope.go:122] "RemoveContainer" containerID="a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d" Oct 30 13:22:09.028672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:09.028628 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:09.028843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:09.028717 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:09.028843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:09.028762 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:09.034695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:09.034619 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:22:09.097155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:09.097124 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:10.027375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:10.027336 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" event={"ID":"d6f09afe-2478-4f1c-b9c0-dca74dc14765","Type":"ContainerStarted","Data":"963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7"} Oct 30 13:22:10.550102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:10.550063 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:22:10.551614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:10.551586 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:22:10.551768 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:10.551644 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:22:10.749704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:10.749659 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:11.550479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:11.550440 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:22:11.550912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:11.550498 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:22:12.423913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:12.423852 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:12.550545 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:12.550510 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:22:12.550933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:12.550567 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:22:12.642097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:12.642066 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:13.550004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:13.549970 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:22:13.550202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:13.550022 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:22:14.163954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:14.163914 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:14.550948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:14.550869 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:22:14.550948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:14.550924 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:22:15.556741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:15.556697 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 13:22:15.557344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:15.556764 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 13:22:15.874345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:15.874232 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:16.089490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:16.089454 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:16.672175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:16.671967 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:22:16.673586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:16.673563 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:22:16.674367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:16.674346 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:22:16.994506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:16.994429 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:22:17.388262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:17.387866 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:17.995064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:17.995033 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:18.623295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:18.622484 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:18.858792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:18.858756 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:19.081718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:19.081686 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:19.365515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:19.365431 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:19.680637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:19.680605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:19.952987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:19.952898 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:20.165847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:20.165805 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:20.390060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:20.384193 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:21.655098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.655047 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:21.792166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.792136 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29"] Oct 30 13:22:21.811298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.804133 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:21.824458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.824417 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29"] Oct 30 13:22:21.828968 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.828940 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8740d4\"" Oct 30 13:22:21.829177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.828963 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"routing-overlay\"" Oct 30 13:22:21.829411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.829022 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"kube-root-ca.crt\"" Oct 30 13:22:21.829411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.829102 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"openshift-service-ca.crt\"" Oct 30 13:22:21.868595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.868566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:21.913929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.913697 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-091b9769f7e9\"" Oct 30 13:22:21.913929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.913843 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-e2e--77992003-dockercfg-zp5z8\"" Oct 30 13:22:21.922044 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.922003 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:21.922368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.922343 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:21.928976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.928945 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:21.929293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:21.929244 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdrv\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-kube-api-access-ctdrv\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.049638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.049595 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdrv\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-kube-api-access-ctdrv\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.050535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.050500 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.052704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.051548 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.052704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.051699 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.052704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.052646 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.053163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.053140 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.055083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.055059 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.078245 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.078205 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdrv\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-kube-api-access-ctdrv\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-688644d29\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.141472 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.141433 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:22.479357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:22.479297 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29"] Oct 30 13:22:22.479992 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:22:22.479969 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cac4d93_cdf4_4e0d_84cd_0cd4cfa07b07.slice/crio-80cf1da0f88df832881d07aa9a08fd9401cb7f1ac6ff41e4f8be7d7ef573b5ca WatchSource:0}: Error finding container 80cf1da0f88df832881d07aa9a08fd9401cb7f1ac6ff41e4f8be7d7ef573b5ca: Status 404 returned error can't find the container with id 80cf1da0f88df832881d07aa9a08fd9401cb7f1ac6ff41e4f8be7d7ef573b5ca Oct 30 13:22:23.221640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:23.221602 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" event={"ID":"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07","Type":"ContainerStarted","Data":"c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5"} Oct 30 13:22:23.222163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:23.221667 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" event={"ID":"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07","Type":"ContainerStarted","Data":"80cf1da0f88df832881d07aa9a08fd9401cb7f1ac6ff41e4f8be7d7ef573b5ca"} Oct 30 13:22:23.222163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:23.221695 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:23.266036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:23.265977 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" podStartSLOduration=2.26596325 podStartE2EDuration="2.26596325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:23.262783811 +0000 UTC m=+2961.389584712" watchObservedRunningTime="2026-10-30 13:22:23.26596325 +0000 UTC m=+2961.392764212" Oct 30 13:22:23.423840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:23.423112 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:25.132067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:25.132025 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:25.347433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:25.347387 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:22:25.347618 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:25.347446 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:22:26.824654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:26.824625 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:28.263483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:28.263449 5515 generic.go:356] "Generic (PLEG): container finished" podID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerID="2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5" exitCode=0 Oct 30 13:22:28.264077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:28.263525 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" event={"ID":"c8b4ca2e-e76e-46b0-b514-a03d9827e132","Type":"ContainerDied","Data":"2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5"} Oct 30 13:22:28.264077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:28.264071 5515 scope.go:122] "RemoveContainer" containerID="2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5" Oct 30 13:22:28.265837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:28.265806 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:28.265922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:28.265877 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:28.265922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:28.265906 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:22:28.530862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:28.530765 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:28.989166 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:22:28.989123 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:22:29.271944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:29.271861 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" event={"ID":"c8b4ca2e-e76e-46b0-b514-a03d9827e132","Type":"ContainerStarted","Data":"65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e"} Oct 30 13:22:29.961783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:29.961744 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:22:29.962031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:29.962009 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:29.962094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:29.962072 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 13:22:30.206743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:30.206706 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:30.962041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:30.962000 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:30.962571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:30.962058 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 13:22:31.883800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:31.883761 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:31.961780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:31.961742 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:31.961949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:31.961796 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 13:22:32.961569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:32.961535 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:32.961971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:32.961586 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 13:22:33.744980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:33.744923 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:33.961724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:33.961684 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 13:22:33.962131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:33.961748 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 13:22:34.233887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:34.233856 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:22:34.965786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:34.965760 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:22:34.966230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:34.966217 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:22:34.966925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:34.966904 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:22:35.597087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:35.597050 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:37.313882 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:37.313851 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:37.976953 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:37.976917 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:39.066543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:39.066509 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:39.402488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:39.402451 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:39.903111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:39.888696 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:40.120075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:40.120032 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:40.874033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:40.873996 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:42.613377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:42.613257 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:44.355842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:44.355811 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:44.578640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:44.578609 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:44.791723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:44.791691 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:46.141867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.141825 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:46.275888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.275854 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq"] Oct 30 13:22:46.280096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.280078 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:46.296948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.296919 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-88e9f3-dockercfg-qgp5x\"" Oct 30 13:22:46.314796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.314771 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq"] Oct 30 13:22:46.388182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.388146 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:46.388492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.388437 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-88e9f3\"" Oct 30 13:22:46.392002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.391933 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq\" (UID: \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:46.392560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.392536 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szc5\" (UniqueName: \"kubernetes.io/projected/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-kube-api-access-7szc5\") pod \"payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq\" (UID: \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:46.494030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.493998 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq\" (UID: \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:46.494208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.494111 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szc5\" (UniqueName: \"kubernetes.io/projected/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-kube-api-access-7szc5\") pod \"payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq\" (UID: \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:46.494862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.494840 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq\" (UID: \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:46.517804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.517775 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szc5\" (UniqueName: \"kubernetes.io/projected/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-kube-api-access-7szc5\") pod \"payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq\" (UID: \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:46.593324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.593296 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:46.898172 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:46.898108 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq"] Oct 30 13:22:46.898912 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:22:46.898879 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cee96e_e0c8_4f3d_8d2d_6df4243d1469.slice/crio-de24baeae41f993c33bb26d1088e0bc65bd0379e28fb1d176325bff8df133d67 WatchSource:0}: Error finding container de24baeae41f993c33bb26d1088e0bc65bd0379e28fb1d176325bff8df133d67: Status 404 returned error can't find the container with id de24baeae41f993c33bb26d1088e0bc65bd0379e28fb1d176325bff8df133d67 Oct 30 13:22:47.282634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:47.282533 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:47.428087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:47.428045 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" event={"ID":"63cee96e-e0c8-4f3d-8d2d-6df4243d1469","Type":"ContainerStarted","Data":"6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46"} Oct 30 13:22:47.428315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:47.428100 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" event={"ID":"63cee96e-e0c8-4f3d-8d2d-6df4243d1469","Type":"ContainerStarted","Data":"de24baeae41f993c33bb26d1088e0bc65bd0379e28fb1d176325bff8df133d67"} Oct 30 13:22:47.428315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:47.428132 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:47.469865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:47.469804 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" podStartSLOduration=1.469785929 podStartE2EDuration="1.469785929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:47.469137534 +0000 UTC m=+2985.595938434" watchObservedRunningTime="2026-10-30 13:22:47.469785929 +0000 UTC m=+2985.596586829" Oct 30 13:22:47.896868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:47.896836 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:48.564583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:48.564546 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:48.776331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:48.776299 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:49.022296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:49.022236 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:49.235939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:49.235907 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:49.496584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:49.496542 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:49.705846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:49.705808 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:49.989967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:49.989931 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:50.208034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:50.208000 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:51.323187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:51.323151 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:51.547741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:51.547681 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:53.209170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:53.209139 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:53.905616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:53.905576 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:22:55.001778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:55.001739 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:56.850193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:56.850160 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:58.438963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:58.438932 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:22:58.644093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:22:58.644058 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:22:58.987916 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:22:58.987879 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:23:00.455129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:00.455099 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:02.235641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:02.235607 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:04.004816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:04.004781 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:04.219668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:04.219635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:05.467846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:05.467815 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:05.710011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:05.709973 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:06.011249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:06.011205 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:06.229743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:06.229710 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:07.504159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:07.504125 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:09.395557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:09.395522 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:09.664490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:09.664405 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:09.885433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:09.885403 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:11.389599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.389569 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:11.500535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.500493 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc"] Oct 30 13:23:11.504162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.504141 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:11.526669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.526642 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc"] Oct 30 13:23:11.534723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.534689 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0d9115-dockercfg-pkw4b\"" Oct 30 13:23:11.540588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.540565 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0d9115\"" Oct 30 13:23:11.641510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.641338 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp6ct\" (UniqueName: \"kubernetes.io/projected/f4f4262a-c772-4592-9224-8a0d4bcb87f4-kube-api-access-cp6ct\") pod \"payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc\" (UID: \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:11.641992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.641968 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f4262a-c772-4592-9224-8a0d4bcb87f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc\" (UID: \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:11.653535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.653455 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:11.744006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.743963 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp6ct\" (UniqueName: \"kubernetes.io/projected/f4f4262a-c772-4592-9224-8a0d4bcb87f4-kube-api-access-cp6ct\") pod \"payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc\" (UID: \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:11.744227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.744026 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f4262a-c772-4592-9224-8a0d4bcb87f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc\" (UID: \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:11.744829 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.744805 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f4262a-c772-4592-9224-8a0d4bcb87f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc\" (UID: \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:11.767625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.767590 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp6ct\" (UniqueName: \"kubernetes.io/projected/f4f4262a-c772-4592-9224-8a0d4bcb87f4-kube-api-access-cp6ct\") pod \"payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc\" (UID: \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:11.818068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:11.818035 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:12.138304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:12.135124 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc"] Oct 30 13:23:12.641637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:12.641605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:12.672580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:12.672542 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" event={"ID":"f4f4262a-c772-4592-9224-8a0d4bcb87f4","Type":"ContainerStarted","Data":"8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8"} Oct 30 13:23:12.672769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:12.672608 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" event={"ID":"f4f4262a-c772-4592-9224-8a0d4bcb87f4","Type":"ContainerStarted","Data":"6cb246c5e87b7065fc7cd91ec4e4e46e09ec1b1694a0cc966d9df2dc7cb1ff15"} Oct 30 13:23:12.672769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:12.672638 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:12.716207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:12.716151 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" podStartSLOduration=1.7161339930000001 podStartE2EDuration="1.716133993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:12.714137876 +0000 UTC m=+3010.840938775" watchObservedRunningTime="2026-10-30 13:23:12.716133993 +0000 UTC m=+3010.842934890" Oct 30 13:23:13.099467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:13.099372 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:14.709133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:14.709099 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:16.387022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:16.386989 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:18.009820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:18.009782 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:19.704341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:19.704306 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:21.417100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:21.417065 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:23.169988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:23.169949 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:23.686425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:23.686396 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:23:24.851011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:24.850975 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:26.359047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.359015 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf"] Oct 30 13:23:26.363256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.363236 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.398666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.398628 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf"] Oct 30 13:23:26.463903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.463870 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-88e9f3-serving-cert\"" Oct 30 13:23:26.465529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.465510 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-maas-api-tls\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.465648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.465627 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5856c551-dc30-4e8f-a684-74c4e564b3a8-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.465743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.465723 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djv4\" (UniqueName: \"kubernetes.io/projected/5856c551-dc30-4e8f-a684-74c4e564b3a8-kube-api-access-6djv4\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.465779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.465770 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-metrics-certs\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.566840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.566799 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-metrics-certs\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.567012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.566939 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-maas-api-tls\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.567068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.567031 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5856c551-dc30-4e8f-a684-74c4e564b3a8-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.567121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.567098 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djv4\" (UniqueName: \"kubernetes.io/projected/5856c551-dc30-4e8f-a684-74c4e564b3a8-kube-api-access-6djv4\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.567695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.567667 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5856c551-dc30-4e8f-a684-74c4e564b3a8-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.569341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.569320 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-metrics-certs\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.569440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.569360 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-maas-api-tls\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.589617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.589561 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:26.592064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.592030 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djv4\" (UniqueName: \"kubernetes.io/projected/5856c551-dc30-4e8f-a684-74c4e564b3a8-kube-api-access-6djv4\") pod \"maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.677504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.677474 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:26.811060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.811024 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:26.995343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:26.995318 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf"] Oct 30 13:23:26.995776 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:23:26.995754 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5856c551_dc30_4e8f_a684_74c4e564b3a8.slice/crio-8964caea71509a40419f058f1fa6e3ea80f759b55c2c40dea6274d5e2f3a752d WatchSource:0}: Error finding container 8964caea71509a40419f058f1fa6e3ea80f759b55c2c40dea6274d5e2f3a752d: Status 404 returned error can't find the container with id 8964caea71509a40419f058f1fa6e3ea80f759b55c2c40dea6274d5e2f3a752d Oct 30 13:23:27.347062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:27.346982 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:27.583204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:27.583100 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:27.804812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:27.804773 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" event={"ID":"5856c551-dc30-4e8f-a684-74c4e564b3a8","Type":"ContainerStarted","Data":"20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a"} Oct 30 13:23:27.805013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:27.804837 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" event={"ID":"5856c551-dc30-4e8f-a684-74c4e564b3a8","Type":"ContainerStarted","Data":"8964caea71509a40419f058f1fa6e3ea80f759b55c2c40dea6274d5e2f3a752d"} Oct 30 13:23:27.805013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:27.804862 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:27.865554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:27.865520 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:27.865756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:27.865632 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" podStartSLOduration=1.41811817 podStartE2EDuration="1.865618977s" totalImagesPullingTime="447.500807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:27.86370056 +0000 UTC m=+3025.990501459" watchObservedRunningTime="2026-10-30 13:23:27.865618977 +0000 UTC m=+3025.992419874" Oct 30 13:23:28.394442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:28.394411 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:28.693754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:28.693668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:28.989926 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:23:28.989835 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:23:29.371542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:29.371466 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:29.599192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:29.598414 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:30.299149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:30.299112 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:31.929947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:31.929908 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:33.606407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:33.606377 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:33.817624 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:33.817590 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:23:35.362186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:35.362157 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:37.113607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:37.113570 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:38.714718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:38.714670 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:40.490518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:40.490458 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:42.161130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:42.161090 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:42.448538 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:42.448462 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl"] Oct 30 13:23:42.804608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:42.804528 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:23:44.033845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:44.033809 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:45.717787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:45.717735 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:47.403595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.403558 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:47.449635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.449597 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" podUID="9cbb0f15-db5c-4488-83f7-62d6347c1dd0" containerName="payload-processing" containerID="cri-o://2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb" gracePeriod=25 Oct 30 13:23:47.756412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.756378 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:47.795813 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.795773 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:23:47.853158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.853123 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-routing-overlay-volume\") pod \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " Oct 30 13:23:47.853399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.853207 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-plugins-config-volume\") pod \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " Oct 30 13:23:47.853491 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.853437 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-provider-credentials\") pod \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " Oct 30 13:23:47.853560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.853496 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-kube-api-access-knl9s\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-kube-api-access-knl9s\") pod \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\" (UID: \"9cbb0f15-db5c-4488-83f7-62d6347c1dd0\") " Oct 30 13:23:47.853766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.853743 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-plugins-config-volume" pod "9cbb0f15-db5c-4488-83f7-62d6347c1dd0" (UID: "9cbb0f15-db5c-4488-83f7-62d6347c1dd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:47.854076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.854057 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.854237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.854218 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-routing-overlay-volume" pod "9cbb0f15-db5c-4488-83f7-62d6347c1dd0" (UID: "9cbb0f15-db5c-4488-83f7-62d6347c1dd0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:23:47.860551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.860520 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-provider-credentials" pod "9cbb0f15-db5c-4488-83f7-62d6347c1dd0" (UID: "9cbb0f15-db5c-4488-83f7-62d6347c1dd0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:47.861837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.861809 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-kube-api-access-knl9s" pod "9cbb0f15-db5c-4488-83f7-62d6347c1dd0" (UID: "9cbb0f15-db5c-4488-83f7-62d6347c1dd0"). InnerVolumeSpecName "kube-api-access-knl9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:23:47.955830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.955794 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.955830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.955831 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knl9s\" (UniqueName: \"kubernetes.io/projected/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-kube-api-access-knl9s\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.956083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.955845 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cbb0f15-db5c-4488-83f7-62d6347c1dd0-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:23:47.963682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.963645 5515 generic.go:356] "Generic (PLEG): container finished" podID="9cbb0f15-db5c-4488-83f7-62d6347c1dd0" containerID="2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb" exitCode=0 Oct 30 13:23:47.963869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.963734 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" Oct 30 13:23:47.963869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.963753 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" event={"ID":"9cbb0f15-db5c-4488-83f7-62d6347c1dd0","Type":"ContainerDied","Data":"2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb"} Oct 30 13:23:47.963869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.963832 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl" event={"ID":"9cbb0f15-db5c-4488-83f7-62d6347c1dd0","Type":"ContainerDied","Data":"f2d330b841727648f873a65a53d949e7d6f34132ae6f42ff6b9fb0714d99e2ec"} Oct 30 13:23:47.964051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.963857 5515 scope.go:122] "RemoveContainer" containerID="2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb" Oct 30 13:23:47.975621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.975604 5515 scope.go:122] "RemoveContainer" containerID="2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb" Oct 30 13:23:47.975953 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:23:47.975932 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb\": container with ID starting with 2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb not found: ID does not exist" containerID="2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb" Oct 30 13:23:47.976001 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:47.975962 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb"} err="failed to get container status \"2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb\": rpc error: code = NotFound desc = could not find container \"2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb\": container with ID starting with 2c8b164f24ee9f5476c71478442e055a17830a4a7b7e03b4b8710f9d61c36beb not found: ID does not exist" Oct 30 13:23:48.019433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.019356 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl"] Oct 30 13:23:48.022606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.022585 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4pc2fl"] Oct 30 13:23:48.086657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.086623 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:48.454168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.454135 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs"] Oct 30 13:23:48.455181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.455149 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9cbb0f15-db5c-4488-83f7-62d6347c1dd0" containerName="payload-processing" Oct 30 13:23:48.455181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.455175 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbb0f15-db5c-4488-83f7-62d6347c1dd0" containerName="payload-processing" Oct 30 13:23:48.455416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.455375 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbb0f15-db5c-4488-83f7-62d6347c1dd0" containerName="payload-processing" Oct 30 13:23:48.459438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.459412 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.478690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.478660 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-6e5458768ddf\"" Oct 30 13:23:48.480559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.480527 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs"] Oct 30 13:23:48.565096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.565058 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.565303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.565105 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzqh\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-kube-api-access-smzqh\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.565357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.565322 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.565411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.565386 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.667136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.667099 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.667386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.667155 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzqh\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-kube-api-access-smzqh\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.667386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.667340 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.667386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.667385 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.668221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.668190 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.668375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.668298 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.669973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.669942 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.691446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.691414 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzqh\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-kube-api-access-smzqh\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:48.775026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:48.774922 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:49.083807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:49.083776 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs"] Oct 30 13:23:49.084797 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:23:49.084774 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8ded41_92ff_484b_9bdc_d10559638307.slice/crio-028794c56122a4613d5010e7c5d000a463af88e936063d07a15614251f8eecfe WatchSource:0}: Error finding container 028794c56122a4613d5010e7c5d000a463af88e936063d07a15614251f8eecfe: Status 404 returned error can't find the container with id 028794c56122a4613d5010e7c5d000a463af88e936063d07a15614251f8eecfe Oct 30 13:23:49.293794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:49.293761 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:49.981749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:49.981714 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" event={"ID":"be8ded41-92ff-484b-9bdc-d10559638307","Type":"ContainerStarted","Data":"37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8"} Oct 30 13:23:49.982201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:49.981765 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" event={"ID":"be8ded41-92ff-484b-9bdc-d10559638307","Type":"ContainerStarted","Data":"028794c56122a4613d5010e7c5d000a463af88e936063d07a15614251f8eecfe"} Oct 30 13:23:49.982201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:49.981799 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:23:50.003960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:50.003926 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbb0f15-db5c-4488-83f7-62d6347c1dd0" path="/var/lib/kubelet/pods/9cbb0f15-db5c-4488-83f7-62d6347c1dd0/volumes" Oct 30 13:23:50.025021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:50.024964 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" podStartSLOduration=2.024950939 podStartE2EDuration="2.024950939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:50.023346093 +0000 UTC m=+3048.150146988" watchObservedRunningTime="2026-10-30 13:23:50.024950939 +0000 UTC m=+3048.151751836" Oct 30 13:23:50.906668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:50.906630 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:52.462295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.462230 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt"] Oct 30 13:23:52.467140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.467123 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.496820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.496789 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0d9115-serving-cert\"" Oct 30 13:23:52.501959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.501926 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt"] Oct 30 13:23:52.523704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.523662 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-metrics-certs\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.523864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.523808 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.523930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.523912 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxml\" (UniqueName: \"kubernetes.io/projected/c29663ca-5372-4d33-b444-41ad16ff8cb3-kube-api-access-qhxml\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.524037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.524022 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29663ca-5372-4d33-b444-41ad16ff8cb3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.625944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.625908 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-metrics-certs\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.626124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.625979 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.626124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.626054 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxml\" (UniqueName: \"kubernetes.io/projected/c29663ca-5372-4d33-b444-41ad16ff8cb3-kube-api-access-qhxml\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.626257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.626126 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29663ca-5372-4d33-b444-41ad16ff8cb3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.626836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.626808 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29663ca-5372-4d33-b444-41ad16ff8cb3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.628538 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.628520 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.628609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.628571 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-metrics-certs\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.648748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.648716 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:52.650581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.650561 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxml\" (UniqueName: \"kubernetes.io/projected/c29663ca-5372-4d33-b444-41ad16ff8cb3-kube-api-access-qhxml\") pod \"maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.783407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.783335 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:52.879892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:52.879862 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:53.100425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:53.100401 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt"] Oct 30 13:23:53.101993 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:23:53.101967 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29663ca_5372_4d33_b444_41ad16ff8cb3.slice/crio-424ac7c4dbc6b14cd61aa4573bad2c8cd7b59b41e5cd79fb8c7551a82df4afda WatchSource:0}: Error finding container 424ac7c4dbc6b14cd61aa4573bad2c8cd7b59b41e5cd79fb8c7551a82df4afda: Status 404 returned error can't find the container with id 424ac7c4dbc6b14cd61aa4573bad2c8cd7b59b41e5cd79fb8c7551a82df4afda Oct 30 13:23:54.012621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:54.012585 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" event={"ID":"c29663ca-5372-4d33-b444-41ad16ff8cb3","Type":"ContainerStarted","Data":"7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75"} Oct 30 13:23:54.013091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:54.012632 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" event={"ID":"c29663ca-5372-4d33-b444-41ad16ff8cb3","Type":"ContainerStarted","Data":"424ac7c4dbc6b14cd61aa4573bad2c8cd7b59b41e5cd79fb8c7551a82df4afda"} Oct 30 13:23:54.013091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:54.012725 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:23:54.054996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:54.054945 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" podStartSLOduration=1.729097736 podStartE2EDuration="2.054931713s" totalImagesPullingTime="325.833977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:54.054490391 +0000 UTC m=+3052.181291289" watchObservedRunningTime="2026-10-30 13:23:54.054931713 +0000 UTC m=+3052.181732610" Oct 30 13:23:54.410797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:54.410745 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:56.128831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:56.128794 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:57.925989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:57.925951 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:23:58.989591 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:23:58.989553 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:23:59.588544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:23:59.588513 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:00.024238 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:00.024209 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:24:00.992160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:00.992125 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:24:01.038678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:01.038635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:01.073307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:01.073256 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29"] Oct 30 13:24:01.252971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:01.252871 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:03.005182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:03.005151 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:04.677303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:04.677250 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:06.073648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.073588 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" podUID="6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" containerName="payload-processing" containerID="cri-o://c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5" gracePeriod=25 Oct 30 13:24:06.342851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.342823 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:06.379182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.379158 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:24:06.450360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.450331 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-plugins-config-volume\") pod \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " Oct 30 13:24:06.450540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.450521 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-provider-credentials\") pod \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " Oct 30 13:24:06.450599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.450561 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-routing-overlay-volume\") pod \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " Oct 30 13:24:06.450735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.450715 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-kube-api-access-ctdrv\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-kube-api-access-ctdrv\") pod \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\" (UID: \"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07\") " Oct 30 13:24:06.450831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.450808 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-plugins-config-volume" pod "6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" (UID: "6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:06.451054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.451023 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-routing-overlay-volume" pod "6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" (UID: "6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:06.451377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.451355 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:24:06.451514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.451379 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:24:06.453087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.453057 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-kube-api-access-ctdrv" pod "6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" (UID: "6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07"). InnerVolumeSpecName "kube-api-access-ctdrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:06.453294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.453244 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-provider-credentials" pod "6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" (UID: "6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:06.552352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.552320 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:24:06.552352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:06.552349 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdrv\" (UniqueName: \"kubernetes.io/projected/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07-kube-api-access-ctdrv\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:24:07.107628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:07.107590 5515 generic.go:356] "Generic (PLEG): container finished" podID="6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" containerID="c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5" exitCode=0 Oct 30 13:24:07.108137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:07.107638 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" Oct 30 13:24:07.108137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:07.107675 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" event={"ID":"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07","Type":"ContainerDied","Data":"c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5"} Oct 30 13:24:07.108137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:07.107727 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29" event={"ID":"6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07","Type":"ContainerDied","Data":"80cf1da0f88df832881d07aa9a08fd9401cb7f1ac6ff41e4f8be7d7ef573b5ca"} Oct 30 13:24:07.108137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:07.107748 5515 scope.go:122] "RemoveContainer" containerID="c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5" Oct 30 13:24:07.118757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:07.118740 5515 scope.go:122] "RemoveContainer" containerID="c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5" Oct 30 13:24:07.119016 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:24:07.119000 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5\": container with ID starting with c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5 not found: ID does not exist" containerID="c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5" Oct 30 13:24:07.119070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:07.119023 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5"} err="failed to get container status \"c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5\": rpc error: code = NotFound desc = could not find container \"c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5\": container with ID starting with c6599866277d15c2614de803f128d2f753f63aa021a4b53cf1a4d776752d9aa5 not found: ID does not exist" Oct 30 13:24:07.161158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:07.161134 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29"] Oct 30 13:24:07.173475 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:07.173452 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-688644d29"] Oct 30 13:24:08.016280 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:08.016240 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" path="/var/lib/kubelet/pods/6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07/volumes" Oct 30 13:24:08.114873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:08.114838 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:09.806071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:09.806032 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:11.679206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:11.679173 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:12.743295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:12.743242 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:13.504724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:13.504692 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:15.251229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:15.251197 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:16.652695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:16.652662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:17.147003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:17.146972 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:17.975658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:17.975623 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:18.183278 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:18.183235 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:18.390725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:18.390691 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:18.641970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:18.641884 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:18.849908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:18.849851 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:19.059028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:19.058948 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:19.518048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:19.518016 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:19.736300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:19.734476 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:20.784512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:20.784480 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:21.006263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:21.006228 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:21.418106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:21.418075 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:24:22.566811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:22.566780 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:24.336708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:24.336651 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:25.066594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:25.066559 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:26.211767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:26.211730 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:26.430471 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:26.430254 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:27.235295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:27.235242 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:28.113235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.113199 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:28.866820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.866790 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl"] Oct 30 13:24:28.867586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.867570 5515 cpu_manager.go:414] "containerMap: removing container" podUID="6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" containerName="payload-processing" Oct 30 13:24:28.867648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.867588 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" containerName="payload-processing" Oct 30 13:24:28.867732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.867723 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cac4d93-cdf4-4e0d-84cd-0cd4cfa07b07" containerName="payload-processing" Oct 30 13:24:28.873067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.873042 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:28.892679 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.892652 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl"] Oct 30 13:24:28.900416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.900391 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-f0f7bfc91557\"" Oct 30 13:24:28.900577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.900561 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"routing-overlay\"" Oct 30 13:24:28.900645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.900631 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"kube-root-ca.crt\"" Oct 30 13:24:28.900704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.900670 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"openshift-service-ca.crt\"" Oct 30 13:24:28.904617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.904591 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d2a0b\"" Oct 30 13:24:28.904771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.904663 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-e2e--e20aaa17-dockercfg-2m475\"" Oct 30 13:24:28.904771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.904733 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-42adde31410a\"" Oct 30 13:24:28.904771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.904597 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-11b2d8954595\"" Oct 30 13:24:28.990663 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:24:28.990624 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:24:28.995533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.995505 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxbdr\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-kube-api-access-sxbdr\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:28.995666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.995544 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:28.995712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.995657 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:28.995826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:28.995810 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.096811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.096775 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxbdr\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-kube-api-access-sxbdr\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.096811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.096813 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.097021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.096939 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.097163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.097145 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.097542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.097525 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.097595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.097564 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.099650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.099629 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.120186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.120126 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxbdr\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-kube-api-access-sxbdr\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.125744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.125716 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:29.187359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.187333 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:29.376391 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.366139 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:29.502169 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.502054 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl"] Oct 30 13:24:29.890795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:29.890759 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:30.110263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:30.110226 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:30.328344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:30.323515 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" event={"ID":"1388c5be-03f7-407f-b80e-3d675d900957","Type":"ContainerStarted","Data":"8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb"} Oct 30 13:24:30.328344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:30.323584 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" event={"ID":"1388c5be-03f7-407f-b80e-3d675d900957","Type":"ContainerStarted","Data":"8d602c184f0012ffef71ad76e06f3975b8d94dd03c6301e1d58777592a063856"} Oct 30 13:24:30.328344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:30.324557 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:30.398095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:30.398024 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" podStartSLOduration=2.398003945 podStartE2EDuration="2.398003945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:30.393534404 +0000 UTC m=+3088.520335309" watchObservedRunningTime="2026-10-30 13:24:30.398003945 +0000 UTC m=+3088.524804844" Oct 30 13:24:31.893232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:31.893200 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:33.816978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:33.816946 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:34.830436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:34.830397 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:35.054103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:35.054039 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:35.262794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:35.262761 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:35.533710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:35.533616 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:36.205246 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:36.204000 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:36.429488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:36.429453 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:36.636814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:36.636769 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:37.435437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:37.435397 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:38.769872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:38.769822 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:39.002384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:39.002351 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:39.431631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:39.431593 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:39.714569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:39.714478 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:40.067607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:40.067529 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:40.463111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:40.463075 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs"] Oct 30 13:24:41.173855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:41.173819 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:41.838461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:41.838422 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:42.341895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:42.341866 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:24:42.985157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:42.985128 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:43.225125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:43.225095 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:43.462493 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:43.462457 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:43.799468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:43.799367 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:44.011645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:44.011603 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:44.580016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:44.579972 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:44.799803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:44.799760 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:45.273096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.273053 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:45.463529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.463488 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" podUID="be8ded41-92ff-484b-9bdc-d10559638307" containerName="payload-processing" containerID="cri-o://37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8" gracePeriod=25 Oct 30 13:24:45.735476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.735443 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:24:45.850855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.850781 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-plugins-config-volume\") pod \"be8ded41-92ff-484b-9bdc-d10559638307\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " Oct 30 13:24:45.850855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.850824 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-routing-overlay-volume\") pod \"be8ded41-92ff-484b-9bdc-d10559638307\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " Oct 30 13:24:45.851083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.850955 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-kube-api-access-smzqh\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-kube-api-access-smzqh\") pod \"be8ded41-92ff-484b-9bdc-d10559638307\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " Oct 30 13:24:45.851083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.851057 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-provider-credentials\") pod \"be8ded41-92ff-484b-9bdc-d10559638307\" (UID: \"be8ded41-92ff-484b-9bdc-d10559638307\") " Oct 30 13:24:45.851255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.851224 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-routing-overlay-volume" pod "be8ded41-92ff-484b-9bdc-d10559638307" (UID: "be8ded41-92ff-484b-9bdc-d10559638307"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:45.851255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.851245 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-plugins-config-volume" pod "be8ded41-92ff-484b-9bdc-d10559638307" (UID: "be8ded41-92ff-484b-9bdc-d10559638307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:24:45.851702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.851682 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:24:45.851768 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.851708 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/be8ded41-92ff-484b-9bdc-d10559638307-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:24:45.853135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.853111 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-kube-api-access-smzqh" pod "be8ded41-92ff-484b-9bdc-d10559638307" (UID: "be8ded41-92ff-484b-9bdc-d10559638307"). InnerVolumeSpecName "kube-api-access-smzqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:45.853309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.853291 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-provider-credentials" pod "be8ded41-92ff-484b-9bdc-d10559638307" (UID: "be8ded41-92ff-484b-9bdc-d10559638307"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:24:45.953098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.953065 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzqh\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-kube-api-access-smzqh\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:24:45.953098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:45.953094 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/be8ded41-92ff-484b-9bdc-d10559638307-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:24:46.244185 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.244147 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:46.549428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.545526 5515 generic.go:356] "Generic (PLEG): container finished" podID="be8ded41-92ff-484b-9bdc-d10559638307" containerID="37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8" exitCode=0 Oct 30 13:24:46.549428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.545626 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" event={"ID":"be8ded41-92ff-484b-9bdc-d10559638307","Type":"ContainerDied","Data":"37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8"} Oct 30 13:24:46.549428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.545675 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" event={"ID":"be8ded41-92ff-484b-9bdc-d10559638307","Type":"ContainerDied","Data":"028794c56122a4613d5010e7c5d000a463af88e936063d07a15614251f8eecfe"} Oct 30 13:24:46.549428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.545707 5515 scope.go:122] "RemoveContainer" containerID="37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8" Oct 30 13:24:46.549428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.545875 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs" Oct 30 13:24:46.587681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.587568 5515 scope.go:122] "RemoveContainer" containerID="37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8" Oct 30 13:24:46.588046 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:24:46.587952 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8\": container with ID starting with 37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8 not found: ID does not exist" containerID="37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8" Oct 30 13:24:46.588046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.587992 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8"} err="failed to get container status \"37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8\": rpc error: code = NotFound desc = could not find container \"37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8\": container with ID starting with 37e515a4d97f8bf56fbc84ecca743d198d6fff5a273fe353cd0317274396d4b8 not found: ID does not exist" Oct 30 13:24:46.606436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.606387 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs"] Oct 30 13:24:46.618606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:46.618580 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7bx2qs"] Oct 30 13:24:47.224057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:47.224022 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:48.006724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:48.006683 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8ded41-92ff-484b-9bdc-d10559638307" path="/var/lib/kubelet/pods/be8ded41-92ff-484b-9bdc-d10559638307/volumes" Oct 30 13:24:49.106494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:49.106460 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:50.925079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:50.925046 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:52.700290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:52.697517 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:52.934296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:52.934235 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:53.401665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:53.401629 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:54.567681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:54.567630 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:56.296039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:56.295959 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:57.909363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:57.909334 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:24:58.988579 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:24:58.988540 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:24:59.609176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:24:59.609143 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:00.153281 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:00.153241 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:00.371359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:00.371328 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:01.252850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:01.252810 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:02.996492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:02.996457 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:04.779779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:04.779746 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:06.517031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:06.516998 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:06.850804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:06.850694 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:07.074346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:07.074314 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:07.354303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:07.353747 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:07.596305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:07.587071 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:08.237300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.237182 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:08.393850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.393815 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k"] Oct 30 13:25:08.394605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.394589 5515 cpu_manager.go:414] "containerMap: removing container" podUID="be8ded41-92ff-484b-9bdc-d10559638307" containerName="payload-processing" Oct 30 13:25:08.394655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.394606 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8ded41-92ff-484b-9bdc-d10559638307" containerName="payload-processing" Oct 30 13:25:08.394742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.394731 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8ded41-92ff-484b-9bdc-d10559638307" containerName="payload-processing" Oct 30 13:25:08.399948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.399932 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.420709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.420679 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k"] Oct 30 13:25:08.428452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.428393 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"e2e-cmp-legacy-5fc8b3-provider-cred\"" Oct 30 13:25:08.428452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.428401 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"kube-root-ca.crt\"" Oct 30 13:25:08.428452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.428401 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4db150\"" Oct 30 13:25:08.428753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.428460 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-e2e--1b2e61a1-dockercfg-q22td\"" Oct 30 13:25:08.431027 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.431004 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"routing-overlay\"" Oct 30 13:25:08.431182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.431082 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"openshift-service-ca.crt\"" Oct 30 13:25:08.462178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.462145 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:08.471986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.471958 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.472121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.472025 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twsls\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-kube-api-access-twsls\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.472121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.472067 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.472219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.472156 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.581336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.580309 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.581336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.580366 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twsls\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-kube-api-access-twsls\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.581336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.580407 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.581336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.580486 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.581710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.581619 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.587287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.582580 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.605814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.605746 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.609566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.609513 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twsls\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-kube-api-access-twsls\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:08.715599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:08.715556 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:09.031784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:09.031753 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k"] Oct 30 13:25:09.032155 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:25:09.032130 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c67a12f_e233_4512_86c1_c84f0bec5f4a.slice/crio-3b9d22ffb1d59ebcc79f89afdb5bee0a9883a012aed7b3eb30081988b9e5e8d2 WatchSource:0}: Error finding container 3b9d22ffb1d59ebcc79f89afdb5bee0a9883a012aed7b3eb30081988b9e5e8d2: Status 404 returned error can't find the container with id 3b9d22ffb1d59ebcc79f89afdb5bee0a9883a012aed7b3eb30081988b9e5e8d2 Oct 30 13:25:09.712549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:09.712506 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" event={"ID":"1c67a12f-e233-4512-86c1-c84f0bec5f4a","Type":"ContainerStarted","Data":"64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0"} Oct 30 13:25:09.713048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:09.712573 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:09.713048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:09.712595 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" event={"ID":"1c67a12f-e233-4512-86c1-c84f0bec5f4a","Type":"ContainerStarted","Data":"3b9d22ffb1d59ebcc79f89afdb5bee0a9883a012aed7b3eb30081988b9e5e8d2"} Oct 30 13:25:09.757286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:09.757232 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" podStartSLOduration=1.757218487 podStartE2EDuration="1.757218487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:09.754739328 +0000 UTC m=+3127.881540230" watchObservedRunningTime="2026-10-30 13:25:09.757218487 +0000 UTC m=+3127.884019426" Oct 30 13:25:10.225806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:10.225771 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:12.108923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:12.108885 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:12.877595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:12.877562 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:13.806972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:13.806938 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:15.627909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:15.627858 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:17.331498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:17.331465 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:19.254909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:19.254874 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:20.722886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:20.722853 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:25:20.990263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:20.990162 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:22.723919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:22.723872 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:22.933695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:22.933660 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:23.956192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:23.956156 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:25:24.550902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:24.550837 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:26.439170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:26.439131 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:28.392447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:28.392411 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:28.637051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:28.637016 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:28.989561 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:25:28.989526 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:25:30.226688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:30.226657 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:31.924282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:31.924240 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:33.722849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:33.722818 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:33.962072 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:33.962033 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:35.458633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:35.458597 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:35.692725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:35.692691 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:37.221747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:37.221711 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:39.002319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:39.002260 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:40.708337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:40.708307 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:41.973752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:41.973720 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:42.265719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:42.265599 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:42.487488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:42.487449 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:44.202837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:44.202806 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:45.950380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:45.950341 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:46.432247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:46.432209 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl"] Oct 30 13:25:47.532384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:47.532347 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:47.748467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:47.748425 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:49.032116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:49.032074 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:49.400387 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:49.396681 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:51.020858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.020812 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:51.433559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.433517 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" podUID="1388c5be-03f7-407f-b80e-3d675d900957" containerName="payload-processing" containerID="cri-o://8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb" gracePeriod=25 Oct 30 13:25:51.709390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.709364 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:25:51.775777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.775747 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-plugins-config-volume\") pod \"1388c5be-03f7-407f-b80e-3d675d900957\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " Oct 30 13:25:51.775966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.775811 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-kube-api-access-sxbdr\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-kube-api-access-sxbdr\") pod \"1388c5be-03f7-407f-b80e-3d675d900957\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " Oct 30 13:25:51.775966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.775852 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-provider-credentials\") pod \"1388c5be-03f7-407f-b80e-3d675d900957\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " Oct 30 13:25:51.775966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.775929 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-routing-overlay-volume\") pod \"1388c5be-03f7-407f-b80e-3d675d900957\" (UID: \"1388c5be-03f7-407f-b80e-3d675d900957\") " Oct 30 13:25:51.776806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.776766 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-routing-overlay-volume" pod "1388c5be-03f7-407f-b80e-3d675d900957" (UID: "1388c5be-03f7-407f-b80e-3d675d900957"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:51.777095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.777064 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-plugins-config-volume" pod "1388c5be-03f7-407f-b80e-3d675d900957" (UID: "1388c5be-03f7-407f-b80e-3d675d900957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:51.778429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.778405 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-kube-api-access-sxbdr" pod "1388c5be-03f7-407f-b80e-3d675d900957" (UID: "1388c5be-03f7-407f-b80e-3d675d900957"). InnerVolumeSpecName "kube-api-access-sxbdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:51.783748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.783724 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-provider-credentials" pod "1388c5be-03f7-407f-b80e-3d675d900957" (UID: "1388c5be-03f7-407f-b80e-3d675d900957"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:51.877454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.877415 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:25:51.877454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.877443 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxbdr\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-kube-api-access-sxbdr\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:25:51.877454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.877453 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388c5be-03f7-407f-b80e-3d675d900957-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:25:51.877454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:51.877462 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388c5be-03f7-407f-b80e-3d675d900957-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:25:52.075121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.075088 5515 generic.go:356] "Generic (PLEG): container finished" podID="1388c5be-03f7-407f-b80e-3d675d900957" containerID="8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb" exitCode=0 Oct 30 13:25:52.075569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.075147 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" Oct 30 13:25:52.075569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.075165 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" event={"ID":"1388c5be-03f7-407f-b80e-3d675d900957","Type":"ContainerDied","Data":"8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb"} Oct 30 13:25:52.075569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.075222 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl" event={"ID":"1388c5be-03f7-407f-b80e-3d675d900957","Type":"ContainerDied","Data":"8d602c184f0012ffef71ad76e06f3975b8d94dd03c6301e1d58777592a063856"} Oct 30 13:25:52.075569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.075248 5515 scope.go:122] "RemoveContainer" containerID="8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb" Oct 30 13:25:52.085792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.085776 5515 scope.go:122] "RemoveContainer" containerID="8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb" Oct 30 13:25:52.086052 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:25:52.086033 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb\": container with ID starting with 8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb not found: ID does not exist" containerID="8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb" Oct 30 13:25:52.086122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.086059 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb"} err="failed to get container status \"8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb\": rpc error: code = NotFound desc = could not find container \"8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb\": container with ID starting with 8c17f74b1aab65646fc7e0a7d1c0b0e48f28db495ecdca449774e4c63a1a04cb not found: ID does not exist" Oct 30 13:25:52.131149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.131118 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl"] Oct 30 13:25:52.143046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.143022 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c7fc55rl"] Oct 30 13:25:52.789055 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:52.789025 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:54.003635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:54.003603 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1388c5be-03f7-407f-b80e-3d675d900957" path="/var/lib/kubelet/pods/1388c5be-03f7-407f-b80e-3d675d900957/volumes" Oct 30 13:25:54.593871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:54.593830 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:55.119177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:55.119143 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:55.347727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:55.347690 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:55.653809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:55.653778 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:56.337121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:56.337030 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:56.543421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:56.543384 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:57.674322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.674282 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg"] Oct 30 13:25:57.675279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.675247 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1388c5be-03f7-407f-b80e-3d675d900957" containerName="payload-processing" Oct 30 13:25:57.675336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.675281 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1388c5be-03f7-407f-b80e-3d675d900957" containerName="payload-processing" Oct 30 13:25:57.675441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.675430 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1388c5be-03f7-407f-b80e-3d675d900957" containerName="payload-processing" Oct 30 13:25:57.680949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.680926 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.697011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.696986 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"routing-overlay\"" Oct 30 13:25:57.709847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.709788 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d2a0b\"" Oct 30 13:25:57.710213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.710113 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"openshift-service-ca.crt\"" Oct 30 13:25:57.710213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.710138 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"kube-root-ca.crt\"" Oct 30 13:25:57.715309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.715280 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg"] Oct 30 13:25:57.763179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.763144 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.763402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.763331 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.763469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.763446 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmdb\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-kube-api-access-xkmdb\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.763573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.763558 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.767767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.767725 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-f0f7bfc91557\"" Oct 30 13:25:57.767994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.767964 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-29b6e3a661be\"" Oct 30 13:25:57.768081 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.767994 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-e2e--e20aaa17-dockercfg-2m475\"" Oct 30 13:25:57.864961 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.864928 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmdb\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-kube-api-access-xkmdb\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.865151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.865045 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.865202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.865160 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.865311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.865296 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.866015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.865985 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.866147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.866040 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.868175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.868146 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.888792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.888761 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmdb\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-kube-api-access-xkmdb\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:57.995614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:57.995527 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:58.267850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:58.267747 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:25:58.319562 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:58.319330 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg"] Oct 30 13:25:58.989766 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:25:58.989725 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:25:59.162928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:59.162875 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" event={"ID":"1e7f710d-9d9f-4798-9157-6fa79320863d","Type":"ContainerStarted","Data":"94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6"} Oct 30 13:25:59.163137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:59.162951 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:25:59.163137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:59.162974 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" event={"ID":"1e7f710d-9d9f-4798-9157-6fa79320863d","Type":"ContainerStarted","Data":"78f0750b18b250ff53be10ecb499c5f3e44b9d93a5541dee9279b1386466cc1b"} Oct 30 13:25:59.206911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:59.206852 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" podStartSLOduration=2.206837276 podStartE2EDuration="2.206837276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:59.205704132 +0000 UTC m=+3177.332505024" watchObservedRunningTime="2026-10-30 13:25:59.206837276 +0000 UTC m=+3177.333638173" Oct 30 13:25:59.943093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:25:59.943051 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:01.559845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:01.559810 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:01.999195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:01.999157 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:02.328465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:02.328375 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:02.547170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:02.547135 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:02.959880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:02.959829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:03.170785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:03.170751 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:03.564495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:03.564254 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:03.770033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:03.769999 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:04.456041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:04.455980 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:04.666635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:04.666601 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:05.487699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:05.487661 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:07.209505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:07.209469 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:08.953286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:08.953246 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:09.964451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:09.964361 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:10.173095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:10.173064 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:26:10.262946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:10.262825 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:11.000834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:11.000803 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:12.795977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:12.795916 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:13.025407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:13.025369 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:14.430592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:14.430557 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:15.693153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:15.693103 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:16.190411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:16.190378 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:16.398559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:16.398524 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:18.104546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:18.104517 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:19.797948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:19.797911 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:20.006959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:20.006924 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:26:21.690728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:21.690683 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.317374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:22.317331 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.543951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:22.543896 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:22.973458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:22.973410 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:23.503633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:23.503599 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:24.703495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:24.703461 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:24.907962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:24.907931 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:25.215595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:25.215562 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:25.493781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:25.493686 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:25.818647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:25.818566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:26.239994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:26.239964 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:26.476074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:26.476033 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:27.002602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:27.002570 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:28.933133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:28.933049 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:28.991190 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:26:28.991151 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:26:30.130713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:30.130681 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:30.346029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:30.345993 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:30.688240 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:30.688174 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:32.622704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:32.622665 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:34.445619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:34.445584 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:36.131177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:36.131139 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:36.957503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:36.957452 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:37.183589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:37.183550 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:37.420330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:37.420257 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:37.630318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:37.630199 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:37.909639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:37.909593 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:38.136585 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:38.136547 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:38.358262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:38.358162 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:38.605889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:38.605853 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:38.834895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:38.834848 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:39.612285 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:39.612231 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:39.875240 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:39.875115 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:40.096146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:40.095350 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:42.140558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:42.140512 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:43.633736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:43.633698 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:45.265400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:45.265367 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k"] Oct 30 13:26:45.338399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:45.338364 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:45.617535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:45.617445 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:45.876192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:45.876091 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:46.278139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:46.278105 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:46.488333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:46.488302 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:46.707937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:46.707900 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:46.930553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:46.930519 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:47.203002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:47.202957 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:47.462639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:47.462544 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:47.850466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:47.850382 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:48.129375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:48.129253 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:49.303219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:49.303181 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:49.614350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:49.614248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:50.266646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.266609 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" podUID="1c67a12f-e233-4512-86c1-c84f0bec5f4a" containerName="payload-processing" containerID="cri-o://64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0" gracePeriod=25 Oct 30 13:26:50.536303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.536222 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:26:50.537946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.537923 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-kube-api-access-twsls\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-kube-api-access-twsls\") pod \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " Oct 30 13:26:50.538078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.538051 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-routing-overlay-volume\") pod \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " Oct 30 13:26:50.538215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.538193 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-provider-credentials\") pod \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " Oct 30 13:26:50.538368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.538296 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-plugins-config-volume\") pod \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\" (UID: \"1c67a12f-e233-4512-86c1-c84f0bec5f4a\") " Oct 30 13:26:50.538488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.538466 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-routing-overlay-volume" pod "1c67a12f-e233-4512-86c1-c84f0bec5f4a" (UID: "1c67a12f-e233-4512-86c1-c84f0bec5f4a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:50.538811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.538785 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-plugins-config-volume" pod "1c67a12f-e233-4512-86c1-c84f0bec5f4a" (UID: "1c67a12f-e233-4512-86c1-c84f0bec5f4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:50.538929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.538889 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:26:50.540370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.540345 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-kube-api-access-twsls" pod "1c67a12f-e233-4512-86c1-c84f0bec5f4a" (UID: "1c67a12f-e233-4512-86c1-c84f0bec5f4a"). InnerVolumeSpecName "kube-api-access-twsls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:50.540452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.540423 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-provider-credentials" pod "1c67a12f-e233-4512-86c1-c84f0bec5f4a" (UID: "1c67a12f-e233-4512-86c1-c84f0bec5f4a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:50.610307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.608632 5515 generic.go:356] "Generic (PLEG): container finished" podID="1c67a12f-e233-4512-86c1-c84f0bec5f4a" containerID="64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0" exitCode=0 Oct 30 13:26:50.610307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.608729 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" event={"ID":"1c67a12f-e233-4512-86c1-c84f0bec5f4a","Type":"ContainerDied","Data":"64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0"} Oct 30 13:26:50.610307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.608828 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" Oct 30 13:26:50.610307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.608841 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k" event={"ID":"1c67a12f-e233-4512-86c1-c84f0bec5f4a","Type":"ContainerDied","Data":"3b9d22ffb1d59ebcc79f89afdb5bee0a9883a012aed7b3eb30081988b9e5e8d2"} Oct 30 13:26:50.610307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.608891 5515 scope.go:122] "RemoveContainer" containerID="64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0" Oct 30 13:26:50.625080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.625051 5515 scope.go:122] "RemoveContainer" containerID="64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0" Oct 30 13:26:50.626651 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:26:50.626618 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0\": container with ID starting with 64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0 not found: ID does not exist" containerID="64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0" Oct 30 13:26:50.626797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.626660 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0"} err="failed to get container status \"64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0\": rpc error: code = NotFound desc = could not find container \"64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0\": container with ID starting with 64ba913a98d291607ef28a0a006cb889dcd27b91a6dbcc8f3bf32e7c9e67cde0 not found: ID does not exist" Oct 30 13:26:50.639823 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.639792 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:26:50.639823 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.639824 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c67a12f-e233-4512-86c1-c84f0bec5f4a-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:26:50.640025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.639836 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twsls\" (UniqueName: \"kubernetes.io/projected/1c67a12f-e233-4512-86c1-c84f0bec5f4a-kube-api-access-twsls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:26:50.663907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.663872 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k"] Oct 30 13:26:50.667424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:50.667388 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-64f47ct6k"] Oct 30 13:26:51.441676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:51.441644 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:52.002750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:52.002718 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c67a12f-e233-4512-86c1-c84f0bec5f4a" path="/var/lib/kubelet/pods/1c67a12f-e233-4512-86c1-c84f0bec5f4a/volumes" Oct 30 13:26:52.170479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:52.170448 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:52.420534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:52.420497 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:52.653457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:52.653414 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:52.885133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:52.885103 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:53.159704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:53.159623 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:53.482532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:53.482455 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:54.887318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:54.887259 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:55.116498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.116430 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:55.566693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.566653 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:55.719317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.719285 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477"] Oct 30 13:26:55.720365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.720339 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1c67a12f-e233-4512-86c1-c84f0bec5f4a" containerName="payload-processing" Oct 30 13:26:55.720365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.720359 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c67a12f-e233-4512-86c1-c84f0bec5f4a" containerName="payload-processing" Oct 30 13:26:55.720544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.720509 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c67a12f-e233-4512-86c1-c84f0bec5f4a" containerName="payload-processing" Oct 30 13:26:55.725791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.725772 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.756435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.756401 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477"] Oct 30 13:26:55.764466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.764432 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0505b0\"" Oct 30 13:26:55.764613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.764475 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"openshift-service-ca.crt\"" Oct 30 13:26:55.765071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.765044 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"agc-cred-fafbf0960658\"" Oct 30 13:26:55.765196 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.765079 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"kube-root-ca.crt\"" Oct 30 13:26:55.768484 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.768465 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"routing-overlay\"" Oct 30 13:26:55.768638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.768526 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-e2e--d83bf907-dockercfg-cxtkv\"" Oct 30 13:26:55.845344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.845047 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:55.855665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.855405 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjkhp\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-kube-api-access-kjkhp\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.855665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.855567 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.856079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.855749 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.856079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.855926 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.956947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.956906 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.957427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.957075 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.957427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.957206 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjkhp\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-kube-api-access-kjkhp\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.957427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.957260 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.957784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.957757 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.957853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.957839 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:55.959945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:55.959921 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:56.012298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:56.010584 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjkhp\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-kube-api-access-kjkhp\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:56.041684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:56.040533 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:56.401084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:56.401053 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477"] Oct 30 13:26:56.402378 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:26:56.402348 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad084cab_e03d_4300_907d_f241f71be6fa.slice/crio-5679d7f62c3de3f87c6c3aa15d0f4afb4449e5209834623c5498d3e30c3c6871 WatchSource:0}: Error finding container 5679d7f62c3de3f87c6c3aa15d0f4afb4449e5209834623c5498d3e30c3c6871: Status 404 returned error can't find the container with id 5679d7f62c3de3f87c6c3aa15d0f4afb4449e5209834623c5498d3e30c3c6871 Oct 30 13:26:56.657228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:56.657130 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" event={"ID":"ad084cab-e03d-4300-907d-f241f71be6fa","Type":"ContainerStarted","Data":"6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435"} Oct 30 13:26:56.657228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:56.657188 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" event={"ID":"ad084cab-e03d-4300-907d-f241f71be6fa","Type":"ContainerStarted","Data":"5679d7f62c3de3f87c6c3aa15d0f4afb4449e5209834623c5498d3e30c3c6871"} Oct 30 13:26:56.657228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:56.657216 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:26:56.703191 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:56.703142 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" podStartSLOduration=1.703128341 podStartE2EDuration="1.703128341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:56.702718058 +0000 UTC m=+3234.829518969" watchObservedRunningTime="2026-10-30 13:26:56.703128341 +0000 UTC m=+3234.829929239" Oct 30 13:26:57.621147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:57.621112 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:26:58.989470 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:26:58.989437 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:26:59.308316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:26:59.308233 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:01.081113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:01.081077 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:03.070132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:03.070088 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:03.458322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:03.458292 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:04.798352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:04.798316 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:05.344567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:05.344518 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:06.613207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:06.613173 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:07.668778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:07.668747 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:27:08.393472 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:08.393438 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:09.078432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:09.078399 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477"] Oct 30 13:27:09.999121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:09.999084 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:11.706874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:11.706838 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:13.151315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:13.151252 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:13.378609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:13.378576 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:14.079203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.079168 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" podUID="ad084cab-e03d-4300-907d-f241f71be6fa" containerName="payload-processing" containerID="cri-o://6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435" gracePeriod=25 Oct 30 13:27:14.341637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.341568 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:27:14.426686 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.426647 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-plugins-config-volume\") pod \"ad084cab-e03d-4300-907d-f241f71be6fa\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " Oct 30 13:27:14.426865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.426841 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-provider-credentials\") pod \"ad084cab-e03d-4300-907d-f241f71be6fa\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " Oct 30 13:27:14.426934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.426876 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-kube-api-access-kjkhp\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-kube-api-access-kjkhp\") pod \"ad084cab-e03d-4300-907d-f241f71be6fa\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " Oct 30 13:27:14.427082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.427064 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-routing-overlay-volume\") pod \"ad084cab-e03d-4300-907d-f241f71be6fa\" (UID: \"ad084cab-e03d-4300-907d-f241f71be6fa\") " Oct 30 13:27:14.427183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.427127 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-plugins-config-volume" pod "ad084cab-e03d-4300-907d-f241f71be6fa" (UID: "ad084cab-e03d-4300-907d-f241f71be6fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:14.427619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.427598 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:27:14.427711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.427619 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-routing-overlay-volume" pod "ad084cab-e03d-4300-907d-f241f71be6fa" (UID: "ad084cab-e03d-4300-907d-f241f71be6fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:14.429788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.429763 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-kube-api-access-kjkhp" pod "ad084cab-e03d-4300-907d-f241f71be6fa" (UID: "ad084cab-e03d-4300-907d-f241f71be6fa"). InnerVolumeSpecName "kube-api-access-kjkhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:14.429970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.429817 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-provider-credentials" pod "ad084cab-e03d-4300-907d-f241f71be6fa" (UID: "ad084cab-e03d-4300-907d-f241f71be6fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:14.529229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.529200 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad084cab-e03d-4300-907d-f241f71be6fa-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:27:14.529229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.529227 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:27:14.529229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.529236 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjkhp\" (UniqueName: \"kubernetes.io/projected/ad084cab-e03d-4300-907d-f241f71be6fa-kube-api-access-kjkhp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:27:14.790833 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.790801 5515 generic.go:356] "Generic (PLEG): container finished" podID="ad084cab-e03d-4300-907d-f241f71be6fa" containerID="6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435" exitCode=0 Oct 30 13:27:14.791015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.790850 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" Oct 30 13:27:14.791015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.790881 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" event={"ID":"ad084cab-e03d-4300-907d-f241f71be6fa","Type":"ContainerDied","Data":"6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435"} Oct 30 13:27:14.791015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.790938 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477" event={"ID":"ad084cab-e03d-4300-907d-f241f71be6fa","Type":"ContainerDied","Data":"5679d7f62c3de3f87c6c3aa15d0f4afb4449e5209834623c5498d3e30c3c6871"} Oct 30 13:27:14.791015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.790969 5515 scope.go:122] "RemoveContainer" containerID="6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435" Oct 30 13:27:14.802377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.802357 5515 scope.go:122] "RemoveContainer" containerID="6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435" Oct 30 13:27:14.802636 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:27:14.802615 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435\": container with ID starting with 6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435 not found: ID does not exist" containerID="6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435" Oct 30 13:27:14.802700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.802644 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435"} err="failed to get container status \"6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435\": rpc error: code = NotFound desc = could not find container \"6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435\": container with ID starting with 6713d183a48360ce152d0167bd5d7908bd1615756c668638d12556c8aaa39435 not found: ID does not exist" Oct 30 13:27:14.843791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.843760 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477"] Oct 30 13:27:14.848004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.847980 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-579cnd477"] Oct 30 13:27:14.985656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:14.985625 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:16.003368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:16.003337 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad084cab-e03d-4300-907d-f241f71be6fa" path="/var/lib/kubelet/pods/ad084cab-e03d-4300-907d-f241f71be6fa/volumes" Oct 30 13:27:16.771019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:16.770989 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:18.842505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:18.842463 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:20.432481 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:20.432444 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:22.233311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:22.233249 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:24.111926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:24.111875 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:25.837682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:25.837644 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:27.557598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:27.557537 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:28.988658 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:27:28.988614 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:27:29.260473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:29.260398 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:31.026200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:31.026163 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:32.648961 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:32.648927 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:34.101718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:34.101677 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:27:34.310508 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:34.310464 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:36.105149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:36.105110 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:36.550808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:36.550774 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:37.963586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:37.963555 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:38.191523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.191490 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:38.853262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.853227 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm"] Oct 30 13:27:38.854125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.854101 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ad084cab-e03d-4300-907d-f241f71be6fa" containerName="payload-processing" Oct 30 13:27:38.854125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.854124 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad084cab-e03d-4300-907d-f241f71be6fa" containerName="payload-processing" Oct 30 13:27:38.854379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.854364 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad084cab-e03d-4300-907d-f241f71be6fa" containerName="payload-processing" Oct 30 13:27:38.858304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.858284 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:38.887851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.887696 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"kube-root-ca.crt\"" Oct 30 13:27:38.887851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.887755 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-091b9769f7e9\"" Oct 30 13:27:38.887851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.887784 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"openshift-service-ca.crt\"" Oct 30 13:27:38.887851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.887784 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-6e5458768ddf\"" Oct 30 13:27:38.887851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.887824 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"routing-overlay\"" Oct 30 13:27:38.887851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.887852 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8740d4\"" Oct 30 13:27:38.888194 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.887968 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-a4617c99fc7d\"" Oct 30 13:27:38.888194 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.888106 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-e2e--77992003-dockercfg-zp5z8\"" Oct 30 13:27:38.888194 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.888150 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-d8628f05d549\"" Oct 30 13:27:38.891777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.891751 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm"] Oct 30 13:27:38.973163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.973132 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjzh\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-kube-api-access-vsjzh\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:38.973617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.973251 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:38.973617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.973305 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:38.973617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:38.973364 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.074825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.074796 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjzh\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-kube-api-access-vsjzh\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.075002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.074889 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.075002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.074911 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.075138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.075006 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.075710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.075686 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.075835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.075813 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.077730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.077707 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.098881 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.098853 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjzh\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-kube-api-access-vsjzh\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.175543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.175514 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:39.479321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.479292 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm"] Oct 30 13:27:39.480557 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:27:39.480534 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80797bc4_81ce_4991_89b0_536c400a698b.slice/crio-4ac6dbd1e7c801b1e0e47befcc5c29a4d416b3d836a4b0e523d1fa2df725aa06 WatchSource:0}: Error finding container 4ac6dbd1e7c801b1e0e47befcc5c29a4d416b3d836a4b0e523d1fa2df725aa06: Status 404 returned error can't find the container with id 4ac6dbd1e7c801b1e0e47befcc5c29a4d416b3d836a4b0e523d1fa2df725aa06 Oct 30 13:27:39.901620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.901584 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:39.973423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.973393 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" event={"ID":"80797bc4-81ce-4991-89b0-536c400a698b","Type":"ContainerStarted","Data":"9dc218658d11ca5f942894603cf8e459264730c41cd9952937e8f1eae175b520"} Oct 30 13:27:39.973803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.973445 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" event={"ID":"80797bc4-81ce-4991-89b0-536c400a698b","Type":"ContainerStarted","Data":"4ac6dbd1e7c801b1e0e47befcc5c29a4d416b3d836a4b0e523d1fa2df725aa06"} Oct 30 13:27:39.973803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:39.973475 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:40.017723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:40.017678 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" podStartSLOduration=2.017664381 podStartE2EDuration="2.017664381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:40.014822946 +0000 UTC m=+3278.141623843" watchObservedRunningTime="2026-10-30 13:27:40.017664381 +0000 UTC m=+3278.144465279" Oct 30 13:27:41.542566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:41.542536 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:42.178236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:42.178198 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:42.397578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:42.397548 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:43.492369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:43.492334 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:43.723208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:43.723131 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:43.928860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:43.928822 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:45.207022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:45.206984 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:46.822101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:46.822069 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:48.865593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:48.865563 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:49.748256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:49.748204 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:49.957002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:49.956945 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:50.395704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:50.395668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:50.602673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:50.602641 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:50.807326 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:50.807237 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:50.985986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:50.985950 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:27:51.161420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:51.161388 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:52.586469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:52.586440 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:54.197164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:54.197126 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:54.409733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:54.409701 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg"] Oct 30 13:27:55.908843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:55.908809 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:57.126202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:57.126137 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:57.358110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:57.358079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:57.560793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:57.560762 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:58.990625 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:27:58.990593 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:27:59.385698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.385665 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:27:59.410936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.410903 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" podUID="1e7f710d-9d9f-4798-9157-6fa79320863d" containerName="payload-processing" containerID="cri-o://94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6" gracePeriod=25 Oct 30 13:27:59.690244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.690217 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:27:59.781511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.781478 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-kube-api-access-xkmdb\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-kube-api-access-xkmdb\") pod \"1e7f710d-9d9f-4798-9157-6fa79320863d\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " Oct 30 13:27:59.781708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.781538 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-routing-overlay-volume\") pod \"1e7f710d-9d9f-4798-9157-6fa79320863d\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " Oct 30 13:27:59.781793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.781727 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-provider-credentials\") pod \"1e7f710d-9d9f-4798-9157-6fa79320863d\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " Oct 30 13:27:59.781861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.781829 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-plugins-config-volume\") pod \"1e7f710d-9d9f-4798-9157-6fa79320863d\" (UID: \"1e7f710d-9d9f-4798-9157-6fa79320863d\") " Oct 30 13:27:59.781986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.781962 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-routing-overlay-volume" pod "1e7f710d-9d9f-4798-9157-6fa79320863d" (UID: "1e7f710d-9d9f-4798-9157-6fa79320863d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:59.782309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.782262 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-plugins-config-volume" pod "1e7f710d-9d9f-4798-9157-6fa79320863d" (UID: "1e7f710d-9d9f-4798-9157-6fa79320863d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:59.782451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.782361 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:27:59.783888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.783863 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-provider-credentials" pod "1e7f710d-9d9f-4798-9157-6fa79320863d" (UID: "1e7f710d-9d9f-4798-9157-6fa79320863d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:59.783888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.783882 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-kube-api-access-xkmdb" pod "1e7f710d-9d9f-4798-9157-6fa79320863d" (UID: "1e7f710d-9d9f-4798-9157-6fa79320863d"). InnerVolumeSpecName "kube-api-access-xkmdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:59.883521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.883488 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:27:59.883521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.883518 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7f710d-9d9f-4798-9157-6fa79320863d-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:27:59.883521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:27:59.883527 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmdb\" (UniqueName: \"kubernetes.io/projected/1e7f710d-9d9f-4798-9157-6fa79320863d-kube-api-access-xkmdb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:28:00.142310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:00.142260 5515 generic.go:356] "Generic (PLEG): container finished" podID="1e7f710d-9d9f-4798-9157-6fa79320863d" containerID="94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6" exitCode=0 Oct 30 13:28:00.142729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:00.142340 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" Oct 30 13:28:00.142729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:00.142345 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" event={"ID":"1e7f710d-9d9f-4798-9157-6fa79320863d","Type":"ContainerDied","Data":"94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6"} Oct 30 13:28:00.142729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:00.142398 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg" event={"ID":"1e7f710d-9d9f-4798-9157-6fa79320863d","Type":"ContainerDied","Data":"78f0750b18b250ff53be10ecb499c5f3e44b9d93a5541dee9279b1386466cc1b"} Oct 30 13:28:00.142729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:00.142419 5515 scope.go:122] "RemoveContainer" containerID="94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6" Oct 30 13:28:00.152709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:00.152693 5515 scope.go:122] "RemoveContainer" containerID="94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6" Oct 30 13:28:00.152982 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:00.152964 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6\": container with ID starting with 94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6 not found: ID does not exist" containerID="94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6" Oct 30 13:28:00.153052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:00.152991 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6"} err="failed to get container status \"94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6\": rpc error: code = NotFound desc = could not find container \"94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6\": container with ID starting with 94a8717d33acf26c5703820c10ebea511b2911291c2b99dbceea5cc139177ba6 not found: ID does not exist" Oct 30 13:28:00.195843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:00.195812 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg"] Oct 30 13:28:00.207933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:00.207910 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-96678wsjg"] Oct 30 13:28:01.191816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:01.191780 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:02.004210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:02.004180 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e7f710d-9d9f-4798-9157-6fa79320863d" path="/var/lib/kubelet/pods/1e7f710d-9d9f-4798-9157-6fa79320863d/volumes" Oct 30 13:28:02.660092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:02.660050 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:02.877601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:02.877565 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:03.112621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:03.112540 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:04.741164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:04.741132 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:04.997759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:04.997684 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:06.686960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:06.686933 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:08.405373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:08.405337 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:10.110189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:10.110153 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:11.133518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:11.133479 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:11.360595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:11.360562 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:11.738938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:11.738904 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:13.289211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:13.289168 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:13.499527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:13.499488 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:15.211696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:15.211664 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:16.067795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:16.067763 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm"] Oct 30 13:28:16.977965 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:16.977930 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:18.628810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:18.628779 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:20.294922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:20.294885 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:21.068233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.068067 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" podUID="80797bc4-81ce-4991-89b0-536c400a698b" containerName="payload-processing" containerID="cri-o://9dc218658d11ca5f942894603cf8e459264730c41cd9952937e8f1eae175b520" gracePeriod=25 Oct 30 13:28:21.323961 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.323931 5515 generic.go:356] "Generic (PLEG): container finished" podID="80797bc4-81ce-4991-89b0-536c400a698b" containerID="9dc218658d11ca5f942894603cf8e459264730c41cd9952937e8f1eae175b520" exitCode=0 Oct 30 13:28:21.324470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.323983 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" event={"ID":"80797bc4-81ce-4991-89b0-536c400a698b","Type":"ContainerDied","Data":"9dc218658d11ca5f942894603cf8e459264730c41cd9952937e8f1eae175b520"} Oct 30 13:28:21.354939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.354911 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:28:21.471486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.471451 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-kube-api-access-vsjzh\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-kube-api-access-vsjzh\") pod \"80797bc4-81ce-4991-89b0-536c400a698b\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " Oct 30 13:28:21.471688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.471546 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-plugins-config-volume\") pod \"80797bc4-81ce-4991-89b0-536c400a698b\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " Oct 30 13:28:21.471760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.471718 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-routing-overlay-volume\") pod \"80797bc4-81ce-4991-89b0-536c400a698b\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " Oct 30 13:28:21.471820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.471769 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-provider-credentials\") pod \"80797bc4-81ce-4991-89b0-536c400a698b\" (UID: \"80797bc4-81ce-4991-89b0-536c400a698b\") " Oct 30 13:28:21.471883 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.471869 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-plugins-config-volume" pod "80797bc4-81ce-4991-89b0-536c400a698b" (UID: "80797bc4-81ce-4991-89b0-536c400a698b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:21.472123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.472083 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-routing-overlay-volume" pod "80797bc4-81ce-4991-89b0-536c400a698b" (UID: "80797bc4-81ce-4991-89b0-536c400a698b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:21.472528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.472511 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:28:21.472590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.472531 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80797bc4-81ce-4991-89b0-536c400a698b-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:28:21.473873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.473840 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-kube-api-access-vsjzh" pod "80797bc4-81ce-4991-89b0-536c400a698b" (UID: "80797bc4-81ce-4991-89b0-536c400a698b"). InnerVolumeSpecName "kube-api-access-vsjzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:21.474060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.474043 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-provider-credentials" pod "80797bc4-81ce-4991-89b0-536c400a698b" (UID: "80797bc4-81ce-4991-89b0-536c400a698b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:21.574352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.574256 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:28:21.574352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.574301 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsjzh\" (UniqueName: \"kubernetes.io/projected/80797bc4-81ce-4991-89b0-536c400a698b-kube-api-access-vsjzh\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:28:21.994418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:21.994381 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:22.332117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:22.332017 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" event={"ID":"80797bc4-81ce-4991-89b0-536c400a698b","Type":"ContainerDied","Data":"4ac6dbd1e7c801b1e0e47befcc5c29a4d416b3d836a4b0e523d1fa2df725aa06"} Oct 30 13:28:22.332117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:22.332039 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm" Oct 30 13:28:22.332117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:22.332081 5515 scope.go:122] "RemoveContainer" containerID="9dc218658d11ca5f942894603cf8e459264730c41cd9952937e8f1eae175b520" Oct 30 13:28:22.386112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:22.386083 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm"] Oct 30 13:28:22.397775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:22.397749 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4595cm"] Oct 30 13:28:23.628573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:23.628534 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:24.002683 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:24.002654 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80797bc4-81ce-4991-89b0-536c400a698b" path="/var/lib/kubelet/pods/80797bc4-81ce-4991-89b0-536c400a698b/volumes" Oct 30 13:28:25.409351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:25.407293 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:27.059758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:27.059725 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:28.702831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:28.702798 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:28.988918 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:28.988831 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:28:30.500720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:30.500679 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:31.779920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:31.779891 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:28:32.117985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:32.117894 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:33.784459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:33.784423 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:35.473459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:35.473431 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:36.694290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:36.693551 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:36.900114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:36.900079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:37.323614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:37.323575 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:37.556288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:37.546795 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:38.899258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:38.899206 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:40.592935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:40.592906 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:42.387720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:42.387626 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:43.718348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:43.718313 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:43.921388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:43.921354 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:44.129499 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:44.129422 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:45.757154 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:45.757122 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:45.975606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:45.975572 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:46.055890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.055804 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85"] Oct 30 13:28:46.056746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.056718 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1e7f710d-9d9f-4798-9157-6fa79320863d" containerName="payload-processing" Oct 30 13:28:46.056746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.056742 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7f710d-9d9f-4798-9157-6fa79320863d" containerName="payload-processing" Oct 30 13:28:46.056925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.056800 5515 cpu_manager.go:414] "containerMap: removing container" podUID="80797bc4-81ce-4991-89b0-536c400a698b" containerName="payload-processing" Oct 30 13:28:46.056925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.056807 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80797bc4-81ce-4991-89b0-536c400a698b" containerName="payload-processing" Oct 30 13:28:46.057045 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.057029 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7f710d-9d9f-4798-9157-6fa79320863d" containerName="payload-processing" Oct 30 13:28:46.057082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.057060 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="80797bc4-81ce-4991-89b0-536c400a698b" containerName="payload-processing" Oct 30 13:28:46.060920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.060903 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.079023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.079003 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8740d4\"" Oct 30 13:28:46.091674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.091641 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-e2e--77992003-dockercfg-zp5z8\"" Oct 30 13:28:46.091955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.091934 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-6e5458768ddf\"" Oct 30 13:28:46.096145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.096122 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85"] Oct 30 13:28:46.133449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.133420 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.133604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.133480 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.133682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.133624 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.133682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.133645 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk42m\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-kube-api-access-jk42m\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.155872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.155843 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"openshift-service-ca.crt\"" Oct 30 13:28:46.155988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.155845 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"routing-overlay\"" Oct 30 13:28:46.155988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.155858 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"kube-root-ca.crt\"" Oct 30 13:28:46.179153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.179130 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-091b9769f7e9\"" Oct 30 13:28:46.234899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.234871 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.235032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.234917 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.235032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.234985 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.235032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.235001 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk42m\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-kube-api-access-jk42m\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.235550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.235529 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.235757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.235739 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.237546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.237530 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.258548 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.258528 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk42m\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-kube-api-access-jk42m\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.375943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.375838 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:46.703380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:46.703352 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85"] Oct 30 13:28:46.704301 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:28:46.704261 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b175f17_6486_403d_9edd_86e8502ce069.slice/crio-36b9682663375234bebf62d5d3aef1861e74d3173de724d13f48fb7e9beb78d6 WatchSource:0}: Error finding container 36b9682663375234bebf62d5d3aef1861e74d3173de724d13f48fb7e9beb78d6: Status 404 returned error can't find the container with id 36b9682663375234bebf62d5d3aef1861e74d3173de724d13f48fb7e9beb78d6 Oct 30 13:28:47.611517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:47.611474 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" event={"ID":"5b175f17-6486-403d-9edd-86e8502ce069","Type":"ContainerStarted","Data":"cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c"} Oct 30 13:28:47.612190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:47.611551 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" event={"ID":"5b175f17-6486-403d-9edd-86e8502ce069","Type":"ContainerStarted","Data":"36b9682663375234bebf62d5d3aef1861e74d3173de724d13f48fb7e9beb78d6"} Oct 30 13:28:47.612190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:47.611707 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:47.641727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:47.641683 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:47.656931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:47.656873 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" podStartSLOduration=1.656860856 podStartE2EDuration="1.656860856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:47.65357532 +0000 UTC m=+3345.780376218" watchObservedRunningTime="2026-10-30 13:28:47.656860856 +0000 UTC m=+3345.783661754" Oct 30 13:28:49.537345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:49.537311 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:50.834226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:50.834188 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:51.051367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:51.051319 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:51.315111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:51.315079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:51.537006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:51.536929 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:51.778854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:51.778820 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:52.483893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:52.483859 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:52.710493 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:52.710458 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:53.510159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:53.510124 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:53.955382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:53.955350 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:54.114101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.114070 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h"] Oct 30 13:28:54.119187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.119167 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.140189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.140152 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h"] Oct 30 13:28:54.149738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.149703 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"kube-root-ca.crt\"" Oct 30 13:28:54.150349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.150326 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1957dc\"" Oct 30 13:28:54.150889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.150613 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"routing-overlay\"" Oct 30 13:28:54.150889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.150682 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"openshift-service-ca.crt\"" Oct 30 13:28:54.151050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.150750 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"agc-cred-fed1fdea6b03\"" Oct 30 13:28:54.151050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.150840 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-e2e--8217e17a-dockercfg-zk6fn\"" Oct 30 13:28:54.214313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.214189 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:54.257918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.257828 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.258438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.258390 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbrp\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-kube-api-access-bjbrp\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.258673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.258655 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.258982 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.258959 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.360753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.360720 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.360911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.360827 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.360951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.360941 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbrp\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-kube-api-access-bjbrp\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.361003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.360967 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.361714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.361688 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.361714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.361696 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.363247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.363228 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.394101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.394062 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbrp\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-kube-api-access-bjbrp\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.435090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.435059 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:54.497053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.496955 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:54.733828 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.733795 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:54.858319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:54.852729 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h"] Oct 30 13:28:55.046292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:55.043377 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:55.268789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:55.268751 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:55.488157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:55.488120 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:55.671034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:55.671000 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" event={"ID":"29ab6b8c-4133-4c97-94e1-c4c136eefea3","Type":"ContainerStarted","Data":"25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6"} Oct 30 13:28:55.671228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:55.671053 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" event={"ID":"29ab6b8c-4133-4c97-94e1-c4c136eefea3","Type":"ContainerStarted","Data":"93be43765fb1dda079040e5e679ab48b8b1e5ac3411902dec9aafc0f2ac85588"} Oct 30 13:28:55.671228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:55.671213 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:28:55.713461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:55.713411 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" podStartSLOduration=1.7133969900000001 podStartE2EDuration="1.71339699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:55.712315964 +0000 UTC m=+3353.839116863" watchObservedRunningTime="2026-10-30 13:28:55.71339699 +0000 UTC m=+3353.840197916" Oct 30 13:28:55.721557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:55.721524 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:55.722150 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:55.722128 5515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-fed1fdea6b03: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:55.722150 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:55.722150 5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:55.722377 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:55.722214 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials podName:29ab6b8c-4133-4c97-94e1-c4c136eefea3 nodeName:}" failed. No retries permitted until 2026-10-30 13:28:56.222197066 +0000 UTC m=+3354.348997953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3") : secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:56.242394 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:56.242355 5515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-fed1fdea6b03: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:56.242394 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:56.242386 5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:56.242842 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:56.242458 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials podName:29ab6b8c-4133-4c97-94e1-c4c136eefea3 nodeName:}" failed. No retries permitted until 2026-10-30 13:28:57.242437905 +0000 UTC m=+3355.369238801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3") : secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:57.255153 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:57.255120 5515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-fed1fdea6b03: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:57.255153 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:57.255146 5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:57.255621 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:57.255206 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials podName:29ab6b8c-4133-4c97-94e1-c4c136eefea3 nodeName:}" failed. No retries permitted until 2026-10-30 13:28:59.255192212 +0000 UTC m=+3357.381993088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3") : secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:57.494913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:57.494881 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:57.874693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:57.874656 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:58.083498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:58.083469 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:28:58.280594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:58.280563 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85"] Oct 30 13:28:58.287060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:58.284668 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:28:58.988888 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:58.988851 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:28:59.303202 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:59.303124 5515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-fed1fdea6b03: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:59.303202 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:59.303147 5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:59.303202 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:28:59.303203 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials podName:29ab6b8c-4133-4c97-94e1-c4c136eefea3 nodeName:}" failed. No retries permitted until 2026-10-30 13:29:03.303182211 +0000 UTC m=+3361.429983086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3") : secret "agc-cred-fed1fdea6b03" not found Oct 30 13:28:59.415012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:28:59.414981 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:00.503262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:00.503225 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:00.807341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:00.803497 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:01.299083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:01.299052 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:01.985436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:01.985405 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:02.190004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:02.189975 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:03.110001 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.109959 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:03.282219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.282181 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" podUID="5b175f17-6486-403d-9edd-86e8502ce069" containerName="payload-processing" containerID="cri-o://cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c" gracePeriod=25 Oct 30 13:29:03.367814 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:03.367748 5515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-fed1fdea6b03: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:29:03.367814 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:03.367772 5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:29:03.367974 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:03.367833 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials podName:29ab6b8c-4133-4c97-94e1-c4c136eefea3 nodeName:}" failed. No retries permitted until 2026-10-30 13:29:11.367813488 +0000 UTC m=+3369.494614363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3") : secret "agc-cred-fed1fdea6b03" not found Oct 30 13:29:03.560469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.560444 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:29:03.671455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.671425 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-provider-credentials\") pod \"5b175f17-6486-403d-9edd-86e8502ce069\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " Oct 30 13:29:03.671455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.671460 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-plugins-config-volume\") pod \"5b175f17-6486-403d-9edd-86e8502ce069\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " Oct 30 13:29:03.671748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.671730 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-routing-overlay-volume\") pod \"5b175f17-6486-403d-9edd-86e8502ce069\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " Oct 30 13:29:03.671849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.671833 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-kube-api-access-jk42m\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-kube-api-access-jk42m\") pod \"5b175f17-6486-403d-9edd-86e8502ce069\" (UID: \"5b175f17-6486-403d-9edd-86e8502ce069\") " Oct 30 13:29:03.672063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.671945 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-plugins-config-volume" pod "5b175f17-6486-403d-9edd-86e8502ce069" (UID: "5b175f17-6486-403d-9edd-86e8502ce069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:03.672150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.672081 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-routing-overlay-volume" pod "5b175f17-6486-403d-9edd-86e8502ce069" (UID: "5b175f17-6486-403d-9edd-86e8502ce069"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:03.672447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.672426 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:03.672553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.672451 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b175f17-6486-403d-9edd-86e8502ce069-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:03.673818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.673793 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-provider-credentials" pod "5b175f17-6486-403d-9edd-86e8502ce069" (UID: "5b175f17-6486-403d-9edd-86e8502ce069"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:03.673935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.673917 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-kube-api-access-jk42m" pod "5b175f17-6486-403d-9edd-86e8502ce069" (UID: "5b175f17-6486-403d-9edd-86e8502ce069"). InnerVolumeSpecName "kube-api-access-jk42m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:03.735301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.735243 5515 generic.go:356] "Generic (PLEG): container finished" podID="5b175f17-6486-403d-9edd-86e8502ce069" containerID="cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c" exitCode=0 Oct 30 13:29:03.735301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.735294 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" event={"ID":"5b175f17-6486-403d-9edd-86e8502ce069","Type":"ContainerDied","Data":"cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c"} Oct 30 13:29:03.735534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.735331 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" Oct 30 13:29:03.735534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.735345 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85" event={"ID":"5b175f17-6486-403d-9edd-86e8502ce069","Type":"ContainerDied","Data":"36b9682663375234bebf62d5d3aef1861e74d3173de724d13f48fb7e9beb78d6"} Oct 30 13:29:03.735534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.735368 5515 scope.go:122] "RemoveContainer" containerID="cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c" Oct 30 13:29:03.747995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.747966 5515 scope.go:122] "RemoveContainer" containerID="cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c" Oct 30 13:29:03.748303 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:03.748257 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c\": container with ID starting with cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c not found: ID does not exist" containerID="cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c" Oct 30 13:29:03.748410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.748311 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c"} err="failed to get container status \"cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c\": rpc error: code = NotFound desc = could not find container \"cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c\": container with ID starting with cbcebcf5e5468ccd18e851898fee3a1bc0abda011a1199993da87f8a90ded23c not found: ID does not exist" Oct 30 13:29:03.773242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.773218 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk42m\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-kube-api-access-jk42m\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:03.773242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.773241 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5b175f17-6486-403d-9edd-86e8502ce069-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:03.789104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.789075 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85"] Oct 30 13:29:03.792459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:03.792438 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-54f7ncl85"] Oct 30 13:29:04.002742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:04.002656 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b175f17-6486-403d-9edd-86e8502ce069" path="/var/lib/kubelet/pods/5b175f17-6486-403d-9edd-86e8502ce069/volumes" Oct 30 13:29:04.793210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:04.793171 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:06.516350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:06.516317 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:06.682344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:06.682316 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:29:08.072871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:08.072840 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h"] Oct 30 13:29:08.215963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:08.215927 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:08.521904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:08.521868 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:08.724023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:08.723994 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:09.520421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:09.520385 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:09.786031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:09.785942 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:10.035652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:10.035620 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:10.345290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:10.344797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:10.551617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:10.551583 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:10.755657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:10.755623 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:11.003222 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:11.003196 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:11.389197 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:11.389163 5515 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1957dc/agc-cred-fed1fdea6b03: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:29:11.389197 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:11.389192 5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h: secret "agc-cred-fed1fdea6b03" not found Oct 30 13:29:11.389458 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:11.389259 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials podName:29ab6b8c-4133-4c97-94e1-c4c136eefea3 nodeName:}" failed. No retries permitted until 2026-10-30 13:29:27.389238564 +0000 UTC m=+3385.516039458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3") : secret "agc-cred-fed1fdea6b03" not found Oct 30 13:29:11.949229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:11.949196 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:12.116917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.116882 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt"] Oct 30 13:29:12.117867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.117842 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5b175f17-6486-403d-9edd-86e8502ce069" containerName="payload-processing" Oct 30 13:29:12.117867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.117864 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b175f17-6486-403d-9edd-86e8502ce069" containerName="payload-processing" Oct 30 13:29:12.118064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.118026 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b175f17-6486-403d-9edd-86e8502ce069" containerName="payload-processing" Oct 30 13:29:12.123796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.123775 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.142239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.142217 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 13:29:12.148493 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.148465 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt"] Oct 30 13:29:12.157978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.157737 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mfkdm\"" Oct 30 13:29:12.157978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.157805 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 13:29:12.157978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.157845 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3c6ac478ec12\"" Oct 30 13:29:12.158294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.158081 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 13:29:12.158562 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.158542 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 13:29:12.159088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.159064 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:12.301637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.301567 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctt2c\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-kube-api-access-ctt2c\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.301797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.301636 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-plugins-config-volume\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.301851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.301829 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-routing-overlay-volume\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.301896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.301889 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-provider-credentials\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.407338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.407141 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-plugins-config-volume\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.407338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.407258 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-routing-overlay-volume\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.407338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.407345 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-provider-credentials\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.407728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.407463 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctt2c\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-kube-api-access-ctt2c\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.409871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.408833 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-plugins-config-volume\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.409871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.409473 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-routing-overlay-volume\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.416443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.414494 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-provider-credentials\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.440016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.439975 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctt2c\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-kube-api-access-ctt2c\") pod \"payload-processing-external-model-797f94cb56-9wjnt\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.440662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.440634 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:12.754913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.754883 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt"] Oct 30 13:29:12.756229 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:29:12.756204 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e09558d_7a30_4a78_ac5a_aaa6691644da.slice/crio-0891b46cf14ddc7bb0f156f9575e4b30eaa96a05438c5512e2c56275c5a636d4 WatchSource:0}: Error finding container 0891b46cf14ddc7bb0f156f9575e4b30eaa96a05438c5512e2c56275c5a636d4: Status 404 returned error can't find the container with id 0891b46cf14ddc7bb0f156f9575e4b30eaa96a05438c5512e2c56275c5a636d4 Oct 30 13:29:12.822602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:12.822572 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" event={"ID":"3e09558d-7a30-4a78-ac5a-aaa6691644da","Type":"ContainerStarted","Data":"0891b46cf14ddc7bb0f156f9575e4b30eaa96a05438c5512e2c56275c5a636d4"} Oct 30 13:29:13.073572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.073485 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" podUID="29ab6b8c-4133-4c97-94e1-c4c136eefea3" containerName="payload-processing" containerID="cri-o://25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6" gracePeriod=25 Oct 30 13:29:13.328181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.328121 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:29:13.421842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.421812 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-kube-api-access-bjbrp\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-kube-api-access-bjbrp\") pod \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " Oct 30 13:29:13.422011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.421889 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-routing-overlay-volume\") pod \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " Oct 30 13:29:13.422062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.422012 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-plugins-config-volume\") pod \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " Oct 30 13:29:13.422114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.422092 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials\") pod \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\" (UID: \"29ab6b8c-4133-4c97-94e1-c4c136eefea3\") " Oct 30 13:29:13.422346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.422250 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-routing-overlay-volume" pod "29ab6b8c-4133-4c97-94e1-c4c136eefea3" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:13.422527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.422502 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-plugins-config-volume" pod "29ab6b8c-4133-4c97-94e1-c4c136eefea3" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:13.422789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.422769 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:13.422904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.422795 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29ab6b8c-4133-4c97-94e1-c4c136eefea3-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:13.424552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.424527 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials" pod "29ab6b8c-4133-4c97-94e1-c4c136eefea3" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:13.424678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.424601 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-kube-api-access-bjbrp" pod "29ab6b8c-4133-4c97-94e1-c4c136eefea3" (UID: "29ab6b8c-4133-4c97-94e1-c4c136eefea3"). InnerVolumeSpecName "kube-api-access-bjbrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:13.425874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.425705 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:13.524544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.524510 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:13.524544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.524537 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjbrp\" (UniqueName: \"kubernetes.io/projected/29ab6b8c-4133-4c97-94e1-c4c136eefea3-kube-api-access-bjbrp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:13.831179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.831129 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" event={"ID":"3e09558d-7a30-4a78-ac5a-aaa6691644da","Type":"ContainerStarted","Data":"e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff"} Oct 30 13:29:13.831390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.831206 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:13.832642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.832613 5515 generic.go:356] "Generic (PLEG): container finished" podID="29ab6b8c-4133-4c97-94e1-c4c136eefea3" containerID="25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6" exitCode=0 Oct 30 13:29:13.832755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.832646 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" Oct 30 13:29:13.832755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.832697 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" event={"ID":"29ab6b8c-4133-4c97-94e1-c4c136eefea3","Type":"ContainerDied","Data":"25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6"} Oct 30 13:29:13.832755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.832745 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h" event={"ID":"29ab6b8c-4133-4c97-94e1-c4c136eefea3","Type":"ContainerDied","Data":"93be43765fb1dda079040e5e679ab48b8b1e5ac3411902dec9aafc0f2ac85588"} Oct 30 13:29:13.832904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.832772 5515 scope.go:122] "RemoveContainer" containerID="25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6" Oct 30 13:29:13.844560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.844542 5515 scope.go:122] "RemoveContainer" containerID="25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6" Oct 30 13:29:13.844849 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:13.844827 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6\": container with ID starting with 25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6 not found: ID does not exist" containerID="25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6" Oct 30 13:29:13.844912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.844857 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6"} err="failed to get container status \"25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6\": rpc error: code = NotFound desc = could not find container \"25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6\": container with ID starting with 25e47cdf612ef7d4719db4bf0a8284c4c3ea5a738d2e92b7a8349381245e21b6 not found: ID does not exist" Oct 30 13:29:13.907774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.907723 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" podStartSLOduration=1.907709422 podStartE2EDuration="1.907709422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:13.905176949 +0000 UTC m=+3372.031977847" watchObservedRunningTime="2026-10-30 13:29:13.907709422 +0000 UTC m=+3372.034510319" Oct 30 13:29:13.931880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.931835 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:13.943828 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.943796 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h"] Oct 30 13:29:13.947820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:13.947797 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5dd5dbt5h"] Oct 30 13:29:14.003580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:14.003552 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ab6b8c-4133-4c97-94e1-c4c136eefea3" path="/var/lib/kubelet/pods/29ab6b8c-4133-4c97-94e1-c4c136eefea3/volumes" Oct 30 13:29:15.612855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:15.612815 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:17.411180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:17.411141 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:17.692187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:17.692082 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:17.909702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:17.909670 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:19.408042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:19.408014 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:21.194603 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:21.194570 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:22.378837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:22.378806 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:22.616644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:22.616613 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:22.996080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:22.996050 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:23.483528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:23.483495 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:23.713372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:23.713340 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:24.084728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:24.084698 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:24.687747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:24.687715 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:24.846344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:24.846317 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:24.905257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:24.905220 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:25.142662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:25.142632 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:25.346064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:25.346035 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:25.588607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:25.588526 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:26.259381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:26.259345 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:26.475442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:26.475412 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:26.689116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:26.689083 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:27.683247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:27.683215 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:28.265741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:28.265704 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:28.989660 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:28.989624 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:29:30.031824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:30.031793 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:30.223581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:30.223550 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:29:31.105235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:31.105190 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:31.329897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:31.329854 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:31.965530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:31.965501 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:34.041927 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:34.041895 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:35.792337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:35.792300 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:37.584492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:37.584462 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:38.083884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:38.083850 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:38.326097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:38.326048 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:39.139960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:39.139923 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:39.386421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:39.386392 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:39.710507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:39.710478 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:41.456863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:41.456827 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:43.069908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:43.069869 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:46.768440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:46.768405 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:46.977213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:46.977183 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:47.183327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:47.183296 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt"] Oct 30 13:29:47.349440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:47.349381 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:48.443725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:48.443687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:48.678122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:48.678089 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:48.908898 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:48.908864 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:49.184981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.184911 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:49.388858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.388825 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:49.453116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.453017 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7"] Oct 30 13:29:49.454083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.454060 5515 cpu_manager.go:414] "containerMap: removing container" podUID="29ab6b8c-4133-4c97-94e1-c4c136eefea3" containerName="payload-processing" Oct 30 13:29:49.454191 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.454084 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ab6b8c-4133-4c97-94e1-c4c136eefea3" containerName="payload-processing" Oct 30 13:29:49.454249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.454239 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ab6b8c-4133-4c97-94e1-c4c136eefea3" containerName="payload-processing" Oct 30 13:29:49.458280 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.458248 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.490819 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.490790 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"routing-overlay\"" Oct 30 13:29:49.491034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.490794 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-acf9e2\"" Oct 30 13:29:49.491034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.490812 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"openshift-service-ca.crt\"" Oct 30 13:29:49.491249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.490938 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"kube-root-ca.crt\"" Oct 30 13:29:49.500203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.500172 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7"] Oct 30 13:29:49.512979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.512947 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"agc-cred-b572fbd3f0eb\"" Oct 30 13:29:49.512979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.512953 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"agc-cred-f967f1739ca3\"" Oct 30 13:29:49.513163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.513035 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"payload-processing-external-model-e2e--66ed339f-dockercfg-5sgmj\"" Oct 30 13:29:49.621404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.621366 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.621587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.621411 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.621587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.621477 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.621587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.621563 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdnfw\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-kube-api-access-wdnfw\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.722654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.722563 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.722654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.722608 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.722654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.722647 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.722958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.722682 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdnfw\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-kube-api-access-wdnfw\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.723608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.723577 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.723730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.723634 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.725534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.725506 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.748069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.748030 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdnfw\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-kube-api-access-wdnfw\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:49.774714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:49.774686 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:50.081455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:50.081423 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7"] Oct 30 13:29:50.082409 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:29:50.082388 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b42d3ee_e0ec_496e_8004_a74605480045.slice/crio-6fac073cd4dc2c360d1a6c193ee4c67b7ed53393873bbe9647e5ccf6b68803a1 WatchSource:0}: Error finding container 6fac073cd4dc2c360d1a6c193ee4c67b7ed53393873bbe9647e5ccf6b68803a1: Status 404 returned error can't find the container with id 6fac073cd4dc2c360d1a6c193ee4c67b7ed53393873bbe9647e5ccf6b68803a1 Oct 30 13:29:50.110060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:50.110028 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" event={"ID":"7b42d3ee-e0ec-496e-8004-a74605480045","Type":"ContainerStarted","Data":"6fac073cd4dc2c360d1a6c193ee4c67b7ed53393873bbe9647e5ccf6b68803a1"} Oct 30 13:29:50.250860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:50.250821 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:50.471464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:50.465567 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:51.118397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:51.118362 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" event={"ID":"7b42d3ee-e0ec-496e-8004-a74605480045","Type":"ContainerStarted","Data":"1fb19bc0a95f5b28637404c3cee2bf0ad0f9cfbbb89e972f5ffde1de43081052"} Oct 30 13:29:51.118570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:51.118459 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:29:51.164798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:51.164750 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" podStartSLOduration=2.164736994 podStartE2EDuration="2.164736994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:51.161799187 +0000 UTC m=+3409.288600083" watchObservedRunningTime="2026-10-30 13:29:51.164736994 +0000 UTC m=+3409.291537892" Oct 30 13:29:51.416028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:51.415998 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:52.184234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.184195 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" podUID="3e09558d-7a30-4a78-ac5a-aaa6691644da" containerName="payload-processing" containerID="cri-o://e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff" gracePeriod=25 Oct 30 13:29:52.353539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.353510 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:52.475510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.475488 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:52.539111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.539073 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-plugins-config-volume\") pod \"3e09558d-7a30-4a78-ac5a-aaa6691644da\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " Oct 30 13:29:52.539355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.539332 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-kube-api-access-ctt2c\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-kube-api-access-ctt2c\") pod \"3e09558d-7a30-4a78-ac5a-aaa6691644da\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " Oct 30 13:29:52.539480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.539460 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-routing-overlay-volume\") pod \"3e09558d-7a30-4a78-ac5a-aaa6691644da\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " Oct 30 13:29:52.539550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.539516 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-provider-credentials\") pod \"3e09558d-7a30-4a78-ac5a-aaa6691644da\" (UID: \"3e09558d-7a30-4a78-ac5a-aaa6691644da\") " Oct 30 13:29:52.539604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.539541 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-plugins-config-volume" pod "3e09558d-7a30-4a78-ac5a-aaa6691644da" (UID: "3e09558d-7a30-4a78-ac5a-aaa6691644da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:52.540056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.540019 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-routing-overlay-volume" pod "3e09558d-7a30-4a78-ac5a-aaa6691644da" (UID: "3e09558d-7a30-4a78-ac5a-aaa6691644da"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:52.540226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.540206 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:52.540340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.540231 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e09558d-7a30-4a78-ac5a-aaa6691644da-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:52.541960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.541934 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-kube-api-access-ctt2c" pod "3e09558d-7a30-4a78-ac5a-aaa6691644da" (UID: "3e09558d-7a30-4a78-ac5a-aaa6691644da"). InnerVolumeSpecName "kube-api-access-ctt2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:52.542199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.542175 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-provider-credentials" pod "3e09558d-7a30-4a78-ac5a-aaa6691644da" (UID: "3e09558d-7a30-4a78-ac5a-aaa6691644da"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:52.641405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.641366 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctt2c\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-kube-api-access-ctt2c\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:52.641405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:52.641400 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e09558d-7a30-4a78-ac5a-aaa6691644da-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:29:53.133424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:53.133388 5515 generic.go:356] "Generic (PLEG): container finished" podID="3e09558d-7a30-4a78-ac5a-aaa6691644da" containerID="e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff" exitCode=0 Oct 30 13:29:53.133614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:53.133435 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" Oct 30 13:29:53.133614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:53.133473 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" event={"ID":"3e09558d-7a30-4a78-ac5a-aaa6691644da","Type":"ContainerDied","Data":"e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff"} Oct 30 13:29:53.133614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:53.133528 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt" event={"ID":"3e09558d-7a30-4a78-ac5a-aaa6691644da","Type":"ContainerDied","Data":"0891b46cf14ddc7bb0f156f9575e4b30eaa96a05438c5512e2c56275c5a636d4"} Oct 30 13:29:53.133614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:53.133553 5515 scope.go:122] "RemoveContainer" containerID="e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff" Oct 30 13:29:53.144491 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:53.144471 5515 scope.go:122] "RemoveContainer" containerID="e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff" Oct 30 13:29:53.144738 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:53.144722 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff\": container with ID starting with e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff not found: ID does not exist" containerID="e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff" Oct 30 13:29:53.144791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:53.144746 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff"} err="failed to get container status \"e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff\": rpc error: code = NotFound desc = could not find container \"e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff\": container with ID starting with e6444a3ffee30609ccc8d646179cb1aa8171fc9b14749015c445b857827b36ff not found: ID does not exist" Oct 30 13:29:53.269207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:53.269180 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt"] Oct 30 13:29:53.280867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:53.280837 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-797f94cb56-9wjnt"] Oct 30 13:29:54.002175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:54.002139 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e09558d-7a30-4a78-ac5a-aaa6691644da" path="/var/lib/kubelet/pods/3e09558d-7a30-4a78-ac5a-aaa6691644da/volumes" Oct 30 13:29:54.226215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:54.226177 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:55.163297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:55.162590 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:55.378700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:55.378668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:55.604118 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:55.604002 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:56.106721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:56.106690 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:56.354382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:56.354283 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:57.453115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.453077 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:57.620359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.620318 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46"] Oct 30 13:29:57.621175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.621156 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3e09558d-7a30-4a78-ac5a-aaa6691644da" containerName="payload-processing" Oct 30 13:29:57.621262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.621176 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e09558d-7a30-4a78-ac5a-aaa6691644da" containerName="payload-processing" Oct 30 13:29:57.621403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.621390 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e09558d-7a30-4a78-ac5a-aaa6691644da" containerName="payload-processing" Oct 30 13:29:57.626392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.626370 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.648366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.648343 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46"] Oct 30 13:29:57.656438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.656411 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-46a10b0d3d71\"" Oct 30 13:29:57.656578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.656420 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 13:29:57.656578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.656536 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kkrjq\"" Oct 30 13:29:57.656685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.656616 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 13:29:57.658738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.658719 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 13:29:57.658738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.658725 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 13:29:57.716569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.716496 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-plugins-config-volume\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.716569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.716541 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-provider-credentials\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.716810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.716571 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-routing-overlay-volume\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.716810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.716677 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fc7\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-kube-api-access-n5fc7\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.821312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.818732 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-plugins-config-volume\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.821312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.818817 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-provider-credentials\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.821312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.818857 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-routing-overlay-volume\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.821312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.818897 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fc7\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-kube-api-access-n5fc7\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.821312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.820416 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-plugins-config-volume\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.821312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.821014 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-routing-overlay-volume\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.844944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.844875 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-provider-credentials\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.856291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.854330 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fc7\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-kube-api-access-n5fc7\") pod \"payload-processing-external-model-7f64b5b66-rhn46\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.925994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.925957 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7"] Oct 30 13:29:57.931008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.930985 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:57.942475 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.942449 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:57.973834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.972951 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tcq9j\"" Oct 30 13:29:57.985338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:57.980142 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7"] Oct 30 13:29:58.014053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.014016 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:29:58.021920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.021883 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.022100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.022081 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42262685-7312-4ac0-8479-be793dab101c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.022149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.022114 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.022189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.022144 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.022228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.022202 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42262685-7312-4ac0-8479-be793dab101c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.022319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.022303 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.022423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.022343 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.022423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.022405 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.022490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.022435 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2ct\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-kube-api-access-zv2ct\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.123978 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124115 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42262685-7312-4ac0-8479-be793dab101c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124299 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124347 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124402 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42262685-7312-4ac0-8479-be793dab101c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124493 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124477 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124504 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124528 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124621 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124663 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2ct\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-kube-api-access-zv2ct\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124745 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.124948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124921 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.125100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.124976 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.125212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.125138 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42262685-7312-4ac0-8479-be793dab101c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.126751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.126731 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.126848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.126766 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42262685-7312-4ac0-8479-be793dab101c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.147767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.147741 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2ct\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-kube-api-access-zv2ct\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.147863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.147798 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7jlq7\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.246368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.246286 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:58.294453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.294422 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46"] Oct 30 13:29:58.295415 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:29:58.295389 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad844b3_480f_4a40_b443_bd9298688672.slice/crio-4362692bfb21948128b5f1b197754f01f222ac89c98d7191f5ac7ef7afbbb8dd WatchSource:0}: Error finding container 4362692bfb21948128b5f1b197754f01f222ac89c98d7191f5ac7ef7afbbb8dd: Status 404 returned error can't find the container with id 4362692bfb21948128b5f1b197754f01f222ac89c98d7191f5ac7ef7afbbb8dd Oct 30 13:29:58.558228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.558199 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7"] Oct 30 13:29:58.558577 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:29:58.558206 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42262685_7312_4ac0_8479_be793dab101c.slice/crio-d549fb81f94e8a4ca4c2538d65108a7283e6340563f194f51ec8dc196d2dc64e WatchSource:0}: Error finding container d549fb81f94e8a4ca4c2538d65108a7283e6340563f194f51ec8dc196d2dc64e: Status 404 returned error can't find the container with id d549fb81f94e8a4ca4c2538d65108a7283e6340563f194f51ec8dc196d2dc64e Oct 30 13:29:58.561050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.561015 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:58.561164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.561098 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:58.561164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:58.561149 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:29:58.990298 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:29:58.990247 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:29:59.225547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.225509 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" event={"ID":"1ad844b3-480f-4a40-b443-bd9298688672","Type":"ContainerStarted","Data":"401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e"} Oct 30 13:29:59.225746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.225575 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:29:59.225746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.225600 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" event={"ID":"1ad844b3-480f-4a40-b443-bd9298688672","Type":"ContainerStarted","Data":"4362692bfb21948128b5f1b197754f01f222ac89c98d7191f5ac7ef7afbbb8dd"} Oct 30 13:29:59.227128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.227099 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" event={"ID":"42262685-7312-4ac0-8479-be793dab101c","Type":"ContainerStarted","Data":"475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618"} Oct 30 13:29:59.227253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.227155 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" event={"ID":"42262685-7312-4ac0-8479-be793dab101c","Type":"ContainerStarted","Data":"d549fb81f94e8a4ca4c2538d65108a7283e6340563f194f51ec8dc196d2dc64e"} Oct 30 13:29:59.246756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.246704 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:29:59.248631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.248607 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7jlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:29:59.248718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.248663 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 13:29:59.271425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.271383 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" podStartSLOduration=2.271372301 podStartE2EDuration="2.271372301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:59.269130696 +0000 UTC m=+3417.395931599" watchObservedRunningTime="2026-10-30 13:29:59.271372301 +0000 UTC m=+3417.398173197" Oct 30 13:29:59.324301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.324191 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podStartSLOduration=2.324171251 podStartE2EDuration="2.324171251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:59.320899786 +0000 UTC m=+3417.447700687" watchObservedRunningTime="2026-10-30 13:29:59.324171251 +0000 UTC m=+3417.450972152" Oct 30 13:29:59.946132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:29:59.946097 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:00.247589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.247507 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7jlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:30:00.247771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.247583 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 13:30:00.324393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.294894 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7"] Oct 30 13:30:00.333477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.333447 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7"] Oct 30 13:30:00.333754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.333707 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:00.366372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.364070 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w"] Oct 30 13:30:00.369566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.369509 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\"" Oct 30 13:30:00.372755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.372733 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h"] Oct 30 13:30:00.373487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.372975 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:00.378236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.378223 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:00.428366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.428329 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w"] Oct 30 13:30:00.435373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.435339 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h"] Oct 30 13:30:00.448016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.447984 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7"] Oct 30 13:30:00.454332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.454311 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:00.458354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.457065 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89"] Oct 30 13:30:00.461594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.461575 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:00.466344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.465651 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6pfc\" (UniqueName: \"kubernetes.io/projected/aceff121-3e3e-4298-b667-e0b14aade56d-kube-api-access-x6pfc\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h\" (UID: \"aceff121-3e3e-4298-b667-e0b14aade56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:00.466344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.465836 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmn2v\" (UniqueName: \"kubernetes.io/projected/7522726f-7b19-45ef-92eb-802bcb81be87-kube-api-access-wmn2v\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w\" (UID: \"7522726f-7b19-45ef-92eb-802bcb81be87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:00.466344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.465882 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7522726f-7b19-45ef-92eb-802bcb81be87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w\" (UID: \"7522726f-7b19-45ef-92eb-802bcb81be87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:00.466344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.465944 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-zkrq7\" (UID: \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:00.466344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.466182 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjq6\" (UniqueName: \"kubernetes.io/projected/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-kube-api-access-lrjq6\") pod \"maas-api-key-cleanup-29889450-zkrq7\" (UID: \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:00.466344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.466235 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aceff121-3e3e-4298-b667-e0b14aade56d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h\" (UID: \"aceff121-3e3e-4298-b667-e0b14aade56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:00.524360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.523945 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7"] Oct 30 13:30:00.545070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.545041 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89"] Oct 30 13:30:00.567843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.567806 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmn2v\" (UniqueName: \"kubernetes.io/projected/7522726f-7b19-45ef-92eb-802bcb81be87-kube-api-access-wmn2v\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w\" (UID: \"7522726f-7b19-45ef-92eb-802bcb81be87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:00.568037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.567864 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7522726f-7b19-45ef-92eb-802bcb81be87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w\" (UID: \"7522726f-7b19-45ef-92eb-802bcb81be87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:00.568037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.567917 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92ce883-ec8c-47e9-b269-019f5d507d75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7\" (UID: \"c92ce883-ec8c-47e9-b269-019f5d507d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:00.568037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.567959 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-zkrq7\" (UID: \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:00.568037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.568024 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jms4m\" (UniqueName: \"kubernetes.io/projected/52db1b88-ea91-4642-93e4-40043dddd4fc-kube-api-access-jms4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89\" (UID: \"52db1b88-ea91-4642-93e4-40043dddd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:00.568362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.568130 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjq6\" (UniqueName: \"kubernetes.io/projected/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-kube-api-access-lrjq6\") pod \"maas-api-key-cleanup-29889450-zkrq7\" (UID: \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:00.568362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.568173 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aceff121-3e3e-4298-b667-e0b14aade56d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h\" (UID: \"aceff121-3e3e-4298-b667-e0b14aade56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:00.568362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.568294 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52db1b88-ea91-4642-93e4-40043dddd4fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89\" (UID: \"52db1b88-ea91-4642-93e4-40043dddd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:00.568362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.568358 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9slpc\" (UniqueName: \"kubernetes.io/projected/c92ce883-ec8c-47e9-b269-019f5d507d75-kube-api-access-9slpc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7\" (UID: \"c92ce883-ec8c-47e9-b269-019f5d507d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:00.568604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.568399 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6pfc\" (UniqueName: \"kubernetes.io/projected/aceff121-3e3e-4298-b667-e0b14aade56d-kube-api-access-x6pfc\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h\" (UID: \"aceff121-3e3e-4298-b667-e0b14aade56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:00.569741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.569665 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aceff121-3e3e-4298-b667-e0b14aade56d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h\" (UID: \"aceff121-3e3e-4298-b667-e0b14aade56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:00.569894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.569869 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7522726f-7b19-45ef-92eb-802bcb81be87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w\" (UID: \"7522726f-7b19-45ef-92eb-802bcb81be87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:00.570019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.570001 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-zkrq7\" (UID: \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:00.623505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.622691 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmn2v\" (UniqueName: \"kubernetes.io/projected/7522726f-7b19-45ef-92eb-802bcb81be87-kube-api-access-wmn2v\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w\" (UID: \"7522726f-7b19-45ef-92eb-802bcb81be87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:00.623505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.623153 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6pfc\" (UniqueName: \"kubernetes.io/projected/aceff121-3e3e-4298-b667-e0b14aade56d-kube-api-access-x6pfc\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h\" (UID: \"aceff121-3e3e-4298-b667-e0b14aade56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:00.624152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.624127 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjq6\" (UniqueName: \"kubernetes.io/projected/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-kube-api-access-lrjq6\") pod \"maas-api-key-cleanup-29889450-zkrq7\" (UID: \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:00.667793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.667764 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:00.669711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.669684 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jms4m\" (UniqueName: \"kubernetes.io/projected/52db1b88-ea91-4642-93e4-40043dddd4fc-kube-api-access-jms4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89\" (UID: \"52db1b88-ea91-4642-93e4-40043dddd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:00.669874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.669820 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52db1b88-ea91-4642-93e4-40043dddd4fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89\" (UID: \"52db1b88-ea91-4642-93e4-40043dddd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:00.669874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.669859 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9slpc\" (UniqueName: \"kubernetes.io/projected/c92ce883-ec8c-47e9-b269-019f5d507d75-kube-api-access-9slpc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7\" (UID: \"c92ce883-ec8c-47e9-b269-019f5d507d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:00.670000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.669950 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92ce883-ec8c-47e9-b269-019f5d507d75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7\" (UID: \"c92ce883-ec8c-47e9-b269-019f5d507d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:00.670473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.670452 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52db1b88-ea91-4642-93e4-40043dddd4fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89\" (UID: \"52db1b88-ea91-4642-93e4-40043dddd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:00.670585 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.670567 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92ce883-ec8c-47e9-b269-019f5d507d75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7\" (UID: \"c92ce883-ec8c-47e9-b269-019f5d507d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:00.690337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.690311 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:00.722293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.722213 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:00.730983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.730748 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9slpc\" (UniqueName: \"kubernetes.io/projected/c92ce883-ec8c-47e9-b269-019f5d507d75-kube-api-access-9slpc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7\" (UID: \"c92ce883-ec8c-47e9-b269-019f5d507d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:00.733407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.733385 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jms4m\" (UniqueName: \"kubernetes.io/projected/52db1b88-ea91-4642-93e4-40043dddd4fc-kube-api-access-jms4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89\" (UID: \"52db1b88-ea91-4642-93e4-40043dddd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:00.785820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.785570 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:00.829119 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:00.827891 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:01.055776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:01.055740 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7"] Oct 30 13:30:01.222842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:01.222814 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89"] Oct 30 13:30:01.247770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:01.247734 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7jlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:30:01.247888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:01.247790 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 13:30:01.248643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:01.248619 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" event={"ID":"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0","Type":"ContainerStarted","Data":"cc799f6fd6e5e6dd12bfdcbc11126bedecfcba6ee1e2b00e799f7909552a83a5"} Oct 30 13:30:01.248752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:01.248671 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7"] Oct 30 13:30:01.326928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:01.326593 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w"] Oct 30 13:30:01.333108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:01.333082 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h"] Oct 30 13:30:02.127017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.126946 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:30:02.247527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.247492 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7jlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:30:02.247778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.247749 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 13:30:02.259048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.259017 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" event={"ID":"aceff121-3e3e-4298-b667-e0b14aade56d","Type":"ContainerStarted","Data":"3a98671c84bde9b1a160a32875ec3669baaee8ad06fb8dc3f9775dcf1143470a"} Oct 30 13:30:02.259200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.259065 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" event={"ID":"aceff121-3e3e-4298-b667-e0b14aade56d","Type":"ContainerStarted","Data":"ef2a8e4b24e9088f4e549feb390aa4a4d19f1302da9e5e5dd1fdf5911aa99838"} Oct 30 13:30:02.260821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.260796 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" event={"ID":"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0","Type":"ContainerStarted","Data":"8e4321993a305a86ba00f7b1f1d6103602327aeb0dfc109205166f8c23af2f41"} Oct 30 13:30:02.263027 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.263001 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" event={"ID":"52db1b88-ea91-4642-93e4-40043dddd4fc","Type":"ContainerStarted","Data":"f77fe02260ab22b5fc18a96af730001e0279371d6d1c441bb56583129cfb83c1"} Oct 30 13:30:02.263106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.263048 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" event={"ID":"52db1b88-ea91-4642-93e4-40043dddd4fc","Type":"ContainerStarted","Data":"5934a424f742b4f56c84249dcd51f9d13aa793e0411b2eeddb2109bfcb40a2cb"} Oct 30 13:30:02.265472 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.265451 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" event={"ID":"c92ce883-ec8c-47e9-b269-019f5d507d75","Type":"ContainerStarted","Data":"97759bc28f3149afcd374d764b0c1de9ac98bfedfc8ae3f39cef9f400aab3041"} Oct 30 13:30:02.265560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.265492 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" event={"ID":"c92ce883-ec8c-47e9-b269-019f5d507d75","Type":"ContainerStarted","Data":"f0803aba186eee5149c5608dbc882c6a9ea8af86c54b7d890a73a96f15068dbb"} Oct 30 13:30:02.267047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.267025 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" event={"ID":"7522726f-7b19-45ef-92eb-802bcb81be87","Type":"ContainerStarted","Data":"2651fa8389b15d7cc12bd019c206005cce3cd0bd258e66ed9d70e87b8c1552f5"} Oct 30 13:30:02.267156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.267061 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" event={"ID":"7522726f-7b19-45ef-92eb-802bcb81be87","Type":"ContainerStarted","Data":"b1d47a9df11a484b1b80bd3c7eee89018121a00cadd7f517de4e64c5b340fbb1"} Oct 30 13:30:02.374240 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.374190 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" podStartSLOduration=2.374175171 podStartE2EDuration="2.374175171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.370322839 +0000 UTC m=+3420.497123770" watchObservedRunningTime="2026-10-30 13:30:02.374175171 +0000 UTC m=+3420.500976066" Oct 30 13:30:02.374452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.374285 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" podStartSLOduration=2.374278101 podStartE2EDuration="2.374278101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.323480579 +0000 UTC m=+3420.450281477" watchObservedRunningTime="2026-10-30 13:30:02.374278101 +0000 UTC m=+3420.501078993" Oct 30 13:30:02.426253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.426191 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" podStartSLOduration=2.426170713 podStartE2EDuration="2.426170713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.423886645 +0000 UTC m=+3420.550687553" watchObservedRunningTime="2026-10-30 13:30:02.426170713 +0000 UTC m=+3420.552971609" Oct 30 13:30:02.466634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.466588 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" podStartSLOduration=2.466575212 podStartE2EDuration="2.466575212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.46421689 +0000 UTC m=+3420.591017787" watchObservedRunningTime="2026-10-30 13:30:02.466575212 +0000 UTC m=+3420.593376109" Oct 30 13:30:02.570081 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.561510 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:02.821926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.821872 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" podStartSLOduration=2.821857904 podStartE2EDuration="2.821857904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.515510525 +0000 UTC m=+3420.642311422" watchObservedRunningTime="2026-10-30 13:30:02.821857904 +0000 UTC m=+3420.948658801" Oct 30 13:30:02.823760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.823732 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46"] Oct 30 13:30:02.920592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:02.920507 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:03.141920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.141885 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:03.249363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.248674 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7jlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:30:03.249363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.248739 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 13:30:03.280754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.279626 5515 generic.go:356] "Generic (PLEG): container finished" podID="aceff121-3e3e-4298-b667-e0b14aade56d" containerID="3a98671c84bde9b1a160a32875ec3669baaee8ad06fb8dc3f9775dcf1143470a" exitCode=60 Oct 30 13:30:03.280754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.279920 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" event={"ID":"aceff121-3e3e-4298-b667-e0b14aade56d","Type":"ContainerDied","Data":"3a98671c84bde9b1a160a32875ec3669baaee8ad06fb8dc3f9775dcf1143470a"} Oct 30 13:30:03.280754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.280150 5515 scope.go:122] "RemoveContainer" containerID="3a98671c84bde9b1a160a32875ec3669baaee8ad06fb8dc3f9775dcf1143470a" Oct 30 13:30:03.284957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.284175 5515 generic.go:356] "Generic (PLEG): container finished" podID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerID="8e4321993a305a86ba00f7b1f1d6103602327aeb0dfc109205166f8c23af2f41" exitCode=60 Oct 30 13:30:03.284957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.284252 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" event={"ID":"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0","Type":"ContainerDied","Data":"8e4321993a305a86ba00f7b1f1d6103602327aeb0dfc109205166f8c23af2f41"} Oct 30 13:30:03.284957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.284645 5515 scope.go:122] "RemoveContainer" containerID="8e4321993a305a86ba00f7b1f1d6103602327aeb0dfc109205166f8c23af2f41" Oct 30 13:30:03.287296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.287226 5515 generic.go:356] "Generic (PLEG): container finished" podID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerID="f77fe02260ab22b5fc18a96af730001e0279371d6d1c441bb56583129cfb83c1" exitCode=60 Oct 30 13:30:03.287417 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.287328 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" event={"ID":"52db1b88-ea91-4642-93e4-40043dddd4fc","Type":"ContainerDied","Data":"f77fe02260ab22b5fc18a96af730001e0279371d6d1c441bb56583129cfb83c1"} Oct 30 13:30:03.288571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.288554 5515 scope.go:122] "RemoveContainer" containerID="f77fe02260ab22b5fc18a96af730001e0279371d6d1c441bb56583129cfb83c1" Oct 30 13:30:03.289058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.289028 5515 generic.go:356] "Generic (PLEG): container finished" podID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerID="97759bc28f3149afcd374d764b0c1de9ac98bfedfc8ae3f39cef9f400aab3041" exitCode=60 Oct 30 13:30:03.289622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.289588 5515 scope.go:122] "RemoveContainer" containerID="97759bc28f3149afcd374d764b0c1de9ac98bfedfc8ae3f39cef9f400aab3041" Oct 30 13:30:03.290221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.289574 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" event={"ID":"c92ce883-ec8c-47e9-b269-019f5d507d75","Type":"ContainerDied","Data":"97759bc28f3149afcd374d764b0c1de9ac98bfedfc8ae3f39cef9f400aab3041"} Oct 30 13:30:03.291637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.291622 5515 generic.go:356] "Generic (PLEG): container finished" podID="7522726f-7b19-45ef-92eb-802bcb81be87" containerID="2651fa8389b15d7cc12bd019c206005cce3cd0bd258e66ed9d70e87b8c1552f5" exitCode=60 Oct 30 13:30:03.291766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.291755 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" event={"ID":"7522726f-7b19-45ef-92eb-802bcb81be87","Type":"ContainerDied","Data":"2651fa8389b15d7cc12bd019c206005cce3cd0bd258e66ed9d70e87b8c1552f5"} Oct 30 13:30:03.292032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:03.292020 5515 scope.go:122] "RemoveContainer" containerID="2651fa8389b15d7cc12bd019c206005cce3cd0bd258e66ed9d70e87b8c1552f5" Oct 30 13:30:04.247446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.247412 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7jlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:30:04.247929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.247461 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 13:30:04.326695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.326661 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" event={"ID":"aceff121-3e3e-4298-b667-e0b14aade56d","Type":"ContainerStarted","Data":"cab58928c32925646b135a18bef9388c263bb030987d3076d72805b9cdb8ba35"} Oct 30 13:30:04.328810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.328787 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" event={"ID":"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0","Type":"ContainerStarted","Data":"45f64a57aa6a52e9282fcfda30d43b7462740e28e3741ff43b3d976ad096cc04"} Oct 30 13:30:04.330434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.330412 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" event={"ID":"52db1b88-ea91-4642-93e4-40043dddd4fc","Type":"ContainerStarted","Data":"156919cca8c42d61f9f5e57df9bae29ee53ac95fda293f50d4807a2d7671ea64"} Oct 30 13:30:04.332816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.332610 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" event={"ID":"c92ce883-ec8c-47e9-b269-019f5d507d75","Type":"ContainerStarted","Data":"116813f42829591c9fae329a706b340e54510b0821827b5642d02fa9da9d64b0"} Oct 30 13:30:04.334311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.334290 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" event={"ID":"7522726f-7b19-45ef-92eb-802bcb81be87","Type":"ContainerStarted","Data":"a9a94f14e62b42ce4a0beb11e50ea33f2898a2febe8ace38906ec30179f58856"} Oct 30 13:30:04.425296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.425137 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:04.657638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.657606 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:04.876640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:04.876548 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:05.091411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.091380 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:05.247540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.247507 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7jlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:30:05.247893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.247568 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 13:30:05.349377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.346380 5515 generic.go:356] "Generic (PLEG): container finished" podID="7522726f-7b19-45ef-92eb-802bcb81be87" containerID="a9a94f14e62b42ce4a0beb11e50ea33f2898a2febe8ace38906ec30179f58856" exitCode=60 Oct 30 13:30:05.349377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.346429 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" event={"ID":"7522726f-7b19-45ef-92eb-802bcb81be87","Type":"ContainerDied","Data":"a9a94f14e62b42ce4a0beb11e50ea33f2898a2febe8ace38906ec30179f58856"} Oct 30 13:30:05.349377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.346506 5515 scope.go:122] "RemoveContainer" containerID="2651fa8389b15d7cc12bd019c206005cce3cd0bd258e66ed9d70e87b8c1552f5" Oct 30 13:30:05.349377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.346856 5515 scope.go:122] "RemoveContainer" containerID="a9a94f14e62b42ce4a0beb11e50ea33f2898a2febe8ace38906ec30179f58856" Oct 30 13:30:05.349377 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:05.347227 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w_odh-ai-gateway-infra(7522726f-7b19-45ef-92eb-802bcb81be87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" Oct 30 13:30:05.349377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.348886 5515 generic.go:356] "Generic (PLEG): container finished" podID="aceff121-3e3e-4298-b667-e0b14aade56d" containerID="cab58928c32925646b135a18bef9388c263bb030987d3076d72805b9cdb8ba35" exitCode=60 Oct 30 13:30:05.349377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.349014 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" event={"ID":"aceff121-3e3e-4298-b667-e0b14aade56d","Type":"ContainerDied","Data":"cab58928c32925646b135a18bef9388c263bb030987d3076d72805b9cdb8ba35"} Oct 30 13:30:05.349377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.349192 5515 scope.go:122] "RemoveContainer" containerID="cab58928c32925646b135a18bef9388c263bb030987d3076d72805b9cdb8ba35" Oct 30 13:30:05.350159 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:05.349584 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h_odh-ai-gateway-infra(aceff121-3e3e-4298-b667-e0b14aade56d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" Oct 30 13:30:05.352523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.350958 5515 generic.go:356] "Generic (PLEG): container finished" podID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerID="45f64a57aa6a52e9282fcfda30d43b7462740e28e3741ff43b3d976ad096cc04" exitCode=60 Oct 30 13:30:05.352523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.351037 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" event={"ID":"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0","Type":"ContainerDied","Data":"45f64a57aa6a52e9282fcfda30d43b7462740e28e3741ff43b3d976ad096cc04"} Oct 30 13:30:05.352523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.351333 5515 scope.go:122] "RemoveContainer" containerID="45f64a57aa6a52e9282fcfda30d43b7462740e28e3741ff43b3d976ad096cc04" Oct 30 13:30:05.352523 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:05.351689 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-zkrq7_odh-ai-gateway-infra(b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" Oct 30 13:30:05.353213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.353191 5515 generic.go:356] "Generic (PLEG): container finished" podID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerID="156919cca8c42d61f9f5e57df9bae29ee53ac95fda293f50d4807a2d7671ea64" exitCode=60 Oct 30 13:30:05.353502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.353488 5515 scope.go:122] "RemoveContainer" containerID="156919cca8c42d61f9f5e57df9bae29ee53ac95fda293f50d4807a2d7671ea64" Oct 30 13:30:05.353616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.353591 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" event={"ID":"52db1b88-ea91-4642-93e4-40043dddd4fc","Type":"ContainerDied","Data":"156919cca8c42d61f9f5e57df9bae29ee53ac95fda293f50d4807a2d7671ea64"} Oct 30 13:30:05.353827 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:05.353805 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89_odh-ai-gateway-infra(52db1b88-ea91-4642-93e4-40043dddd4fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" Oct 30 13:30:05.355602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.355582 5515 generic.go:356] "Generic (PLEG): container finished" podID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerID="116813f42829591c9fae329a706b340e54510b0821827b5642d02fa9da9d64b0" exitCode=60 Oct 30 13:30:05.355692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.355625 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" event={"ID":"c92ce883-ec8c-47e9-b269-019f5d507d75","Type":"ContainerDied","Data":"116813f42829591c9fae329a706b340e54510b0821827b5642d02fa9da9d64b0"} Oct 30 13:30:05.355892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.355877 5515 scope.go:122] "RemoveContainer" containerID="116813f42829591c9fae329a706b340e54510b0821827b5642d02fa9da9d64b0" Oct 30 13:30:05.356254 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:05.356224 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7_odh-ai-gateway-infra(c92ce883-ec8c-47e9-b269-019f5d507d75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" Oct 30 13:30:05.366774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.366759 5515 scope.go:122] "RemoveContainer" containerID="3a98671c84bde9b1a160a32875ec3669baaee8ad06fb8dc3f9775dcf1143470a" Oct 30 13:30:05.381466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.381443 5515 scope.go:122] "RemoveContainer" containerID="8e4321993a305a86ba00f7b1f1d6103602327aeb0dfc109205166f8c23af2f41" Oct 30 13:30:05.395171 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.395147 5515 scope.go:122] "RemoveContainer" containerID="f77fe02260ab22b5fc18a96af730001e0279371d6d1c441bb56583129cfb83c1" Oct 30 13:30:05.410408 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.410388 5515 scope.go:122] "RemoveContainer" containerID="97759bc28f3149afcd374d764b0c1de9ac98bfedfc8ae3f39cef9f400aab3041" Oct 30 13:30:05.877932 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:05.877889 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:06.125291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:06.120748 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:06.247528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:06.247486 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7jlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 13:30:06.247719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:06.247567 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 13:30:06.366337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:06.366306 5515 scope.go:122] "RemoveContainer" containerID="a9a94f14e62b42ce4a0beb11e50ea33f2898a2febe8ace38906ec30179f58856" Oct 30 13:30:06.366717 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:06.366684 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w_odh-ai-gateway-infra(7522726f-7b19-45ef-92eb-802bcb81be87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" Oct 30 13:30:06.368318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:06.368292 5515 scope.go:122] "RemoveContainer" containerID="cab58928c32925646b135a18bef9388c263bb030987d3076d72805b9cdb8ba35" Oct 30 13:30:06.368706 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:06.368675 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h_odh-ai-gateway-infra(aceff121-3e3e-4298-b667-e0b14aade56d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" Oct 30 13:30:06.370286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:06.370255 5515 scope.go:122] "RemoveContainer" containerID="45f64a57aa6a52e9282fcfda30d43b7462740e28e3741ff43b3d976ad096cc04" Oct 30 13:30:06.370580 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:06.370561 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-zkrq7_odh-ai-gateway-infra(b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" Oct 30 13:30:06.372017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:06.371995 5515 scope.go:122] "RemoveContainer" containerID="156919cca8c42d61f9f5e57df9bae29ee53ac95fda293f50d4807a2d7671ea64" Oct 30 13:30:06.372435 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:06.372401 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89_odh-ai-gateway-infra(52db1b88-ea91-4642-93e4-40043dddd4fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" Oct 30 13:30:06.373735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:06.373717 5515 scope.go:122] "RemoveContainer" containerID="116813f42829591c9fae329a706b340e54510b0821827b5642d02fa9da9d64b0" Oct 30 13:30:06.374066 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:06.374042 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7_odh-ai-gateway-infra(c92ce883-ec8c-47e9-b269-019f5d507d75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" Oct 30 13:30:07.296023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:07.295988 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:07.389469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:07.389430 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:30:07.390031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:07.389998 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:30:07.390799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:07.390776 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:30:07.736289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:07.725105 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:07.824912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:07.824870 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" podUID="1ad844b3-480f-4a40-b443-bd9298688672" containerName="payload-processing" containerID="cri-o://401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e" gracePeriod=25 Oct 30 13:30:08.240553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.240529 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:30:08.279036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.279011 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-provider-credentials\") pod \"1ad844b3-480f-4a40-b443-bd9298688672\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " Oct 30 13:30:08.279174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.279065 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-plugins-config-volume\") pod \"1ad844b3-480f-4a40-b443-bd9298688672\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " Oct 30 13:30:08.279174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.279154 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-routing-overlay-volume\") pod \"1ad844b3-480f-4a40-b443-bd9298688672\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " Oct 30 13:30:08.279423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.279401 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-kube-api-access-n5fc7\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-kube-api-access-n5fc7\") pod \"1ad844b3-480f-4a40-b443-bd9298688672\" (UID: \"1ad844b3-480f-4a40-b443-bd9298688672\") " Oct 30 13:30:08.279539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.279492 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-routing-overlay-volume" pod "1ad844b3-480f-4a40-b443-bd9298688672" (UID: "1ad844b3-480f-4a40-b443-bd9298688672"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:08.279616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.279590 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-plugins-config-volume" pod "1ad844b3-480f-4a40-b443-bd9298688672" (UID: "1ad844b3-480f-4a40-b443-bd9298688672"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:08.279973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.279951 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:08.280043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.279978 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ad844b3-480f-4a40-b443-bd9298688672-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:08.281889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.281869 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-provider-credentials" pod "1ad844b3-480f-4a40-b443-bd9298688672" (UID: "1ad844b3-480f-4a40-b443-bd9298688672"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:08.281989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.281936 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-kube-api-access-n5fc7" pod "1ad844b3-480f-4a40-b443-bd9298688672" (UID: "1ad844b3-480f-4a40-b443-bd9298688672"). InnerVolumeSpecName "kube-api-access-n5fc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:08.381116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.381033 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5fc7\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-kube-api-access-n5fc7\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:08.381116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.381063 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ad844b3-480f-4a40-b443-bd9298688672-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:08.389763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.389733 5515 generic.go:356] "Generic (PLEG): container finished" podID="1ad844b3-480f-4a40-b443-bd9298688672" containerID="401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e" exitCode=0 Oct 30 13:30:08.389919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.389796 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" Oct 30 13:30:08.389919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.389816 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" event={"ID":"1ad844b3-480f-4a40-b443-bd9298688672","Type":"ContainerDied","Data":"401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e"} Oct 30 13:30:08.389919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.389884 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46" event={"ID":"1ad844b3-480f-4a40-b443-bd9298688672","Type":"ContainerDied","Data":"4362692bfb21948128b5f1b197754f01f222ac89c98d7191f5ac7ef7afbbb8dd"} Oct 30 13:30:08.389919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.389907 5515 scope.go:122] "RemoveContainer" containerID="401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e" Oct 30 13:30:08.401604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.401498 5515 scope.go:122] "RemoveContainer" containerID="401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e" Oct 30 13:30:08.401842 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:08.401809 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e\": container with ID starting with 401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e not found: ID does not exist" containerID="401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e" Oct 30 13:30:08.401951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.401851 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e"} err="failed to get container status \"401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e\": rpc error: code = NotFound desc = could not find container \"401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e\": container with ID starting with 401039e43c56bb2f1349249e40efec53818c3f24abcfab657cdba1348628cb6e not found: ID does not exist" Oct 30 13:30:08.445102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.445072 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46"] Oct 30 13:30:08.453595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.453566 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-rhn46"] Oct 30 13:30:08.826300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.817150 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:08.971038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.971003 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp"] Oct 30 13:30:08.971832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.971817 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1ad844b3-480f-4a40-b443-bd9298688672" containerName="payload-processing" Oct 30 13:30:08.971908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.971833 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad844b3-480f-4a40-b443-bd9298688672" containerName="payload-processing" Oct 30 13:30:08.971974 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.971962 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad844b3-480f-4a40-b443-bd9298688672" containerName="payload-processing" Oct 30 13:30:08.977094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.977073 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:08.999153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.997224 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4db150\"" Oct 30 13:30:08.999153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.997392 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"agc-cred-32d76723ab73\"" Oct 30 13:30:08.999153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:08.997514 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-e2e--1b2e61a1-dockercfg-6d4mp\"" Oct 30 13:30:09.020853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.020814 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp"] Oct 30 13:30:09.094543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.093621 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.094543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.093702 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.094543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.093820 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6br\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-kube-api-access-db6br\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.094543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.093884 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.097734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.097703 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"routing-overlay\"" Oct 30 13:30:09.097901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.097800 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"kube-root-ca.crt\"" Oct 30 13:30:09.098061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.098036 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"openshift-service-ca.crt\"" Oct 30 13:30:09.209290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.195883 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.209290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.197092 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.209290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.197249 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6br\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-kube-api-access-db6br\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.209290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.197375 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.209290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.198566 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.209290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.199186 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.209290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.200931 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.244020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.241986 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6br\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-kube-api-access-db6br\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.311717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.311681 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:09.506144 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.506083 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:09.635799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:09.635774 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp"] Oct 30 13:30:09.639880 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:30:09.639849 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932ca447_138e_4e3d_80dd_786e0a6cf70d.slice/crio-47ccc0b088bd1e23edb4a9740f9939923841856554d7f983fe2cc8417b741f6e WatchSource:0}: Error finding container 47ccc0b088bd1e23edb4a9740f9939923841856554d7f983fe2cc8417b741f6e: Status 404 returned error can't find the container with id 47ccc0b088bd1e23edb4a9740f9939923841856554d7f983fe2cc8417b741f6e Oct 30 13:30:10.005566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:10.005479 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad844b3-480f-4a40-b443-bd9298688672" path="/var/lib/kubelet/pods/1ad844b3-480f-4a40-b443-bd9298688672/volumes" Oct 30 13:30:10.418772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:10.418729 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" event={"ID":"932ca447-138e-4e3d-80dd-786e0a6cf70d","Type":"ContainerStarted","Data":"5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315"} Oct 30 13:30:10.419003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:10.418795 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" event={"ID":"932ca447-138e-4e3d-80dd-786e0a6cf70d","Type":"ContainerStarted","Data":"47ccc0b088bd1e23edb4a9740f9939923841856554d7f983fe2cc8417b741f6e"} Oct 30 13:30:10.419003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:10.418826 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:10.466120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:10.466067 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" podStartSLOduration=2.466054255 podStartE2EDuration="2.466054255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:10.463577449 +0000 UTC m=+3428.590378345" watchObservedRunningTime="2026-10-30 13:30:10.466054255 +0000 UTC m=+3428.592855201" Oct 30 13:30:11.150620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.150589 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:11.381889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.381831 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:11.577936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.577867 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd"] Oct 30 13:30:11.582468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.582445 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.615840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.615815 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd"] Oct 30 13:30:11.656398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.656362 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.656544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.656404 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6lgq\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-kube-api-access-p6lgq\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.656629 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.656605 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.656697 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.656658 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.757782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.757747 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.757782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.757786 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lgq\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-kube-api-access-p6lgq\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.758037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.757952 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.758037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.758016 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.758674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.758649 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.758831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.758744 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.760733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.760709 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.781698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.781666 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lgq\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-kube-api-access-p6lgq\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:11.898455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:11.898426 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:12.219923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:12.219885 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd"] Oct 30 13:30:12.220851 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:30:12.220819 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86455459_c6d6_4df9_90b6_e9cb8a718fce.slice/crio-3ee5cbfe418334faf8366af75d7bea9385f0d2ca6fa1b49a5e6fc011d7476c4a WatchSource:0}: Error finding container 3ee5cbfe418334faf8366af75d7bea9385f0d2ca6fa1b49a5e6fc011d7476c4a: Status 404 returned error can't find the container with id 3ee5cbfe418334faf8366af75d7bea9385f0d2ca6fa1b49a5e6fc011d7476c4a Oct 30 13:30:12.271121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:12.271092 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd"] Oct 30 13:30:12.274602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:12.274581 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7"] Oct 30 13:30:12.378750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:12.378659 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:12.435644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:12.435605 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" event={"ID":"86455459-c6d6-4df9-90b6-e9cb8a718fce","Type":"ContainerStarted","Data":"d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1"} Oct 30 13:30:12.435806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:12.435667 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" event={"ID":"86455459-c6d6-4df9-90b6-e9cb8a718fce","Type":"ContainerStarted","Data":"3ee5cbfe418334faf8366af75d7bea9385f0d2ca6fa1b49a5e6fc011d7476c4a"} Oct 30 13:30:12.435806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:12.435781 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:12.482938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:12.482841 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" podStartSLOduration=1.482822984 podStartE2EDuration="1.482822984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:12.476552547 +0000 UTC m=+3430.603353445" watchObservedRunningTime="2026-10-30 13:30:12.482822984 +0000 UTC m=+3430.609623882" Oct 30 13:30:12.606894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:12.606857 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:13.027940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.027906 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn"] Oct 30 13:30:13.033244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.033216 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.061110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.061077 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn"] Oct 30 13:30:13.068129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.068085 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:13.081069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.081038 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.081206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.081180 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.081348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.081314 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.081435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.081407 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.081488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.081464 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.081550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.081505 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.081617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.081551 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwchv\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-kube-api-access-lwchv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.081617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.081607 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.081974 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.081952 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183240 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183327 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183388 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183462 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183534 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183581 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183619 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183663 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwchv\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-kube-api-access-lwchv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183685 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183696 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183713 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.183949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.183924 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.184038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.184014 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.184391 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.184367 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.186071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.186048 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.186208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.186191 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.208453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.208432 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.208881 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.208861 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwchv\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-kube-api-access-lwchv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p95xn\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.359208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.359126 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:13.370589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.370559 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:13.584791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.584761 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:13.682149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.682126 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn"] Oct 30 13:30:13.683242 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:30:13.683219 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa52c12b_284b_4a79_90a6_9ed858ed0fb8.slice/crio-fb3bfbd4880a39ea6de24c3fb8476306351c524de33fd3d15c0a4425282a874f WatchSource:0}: Error finding container fb3bfbd4880a39ea6de24c3fb8476306351c524de33fd3d15c0a4425282a874f: Status 404 returned error can't find the container with id fb3bfbd4880a39ea6de24c3fb8476306351c524de33fd3d15c0a4425282a874f Oct 30 13:30:13.686162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.686131 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:13.686227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.686195 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:13.686227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:13.686223 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:30:14.455738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:14.455703 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" event={"ID":"aa52c12b-284b-4a79-90a6-9ed858ed0fb8","Type":"ContainerStarted","Data":"f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1"} Oct 30 13:30:14.456167 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:14.455765 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" event={"ID":"aa52c12b-284b-4a79-90a6-9ed858ed0fb8","Type":"ContainerStarted","Data":"fb3bfbd4880a39ea6de24c3fb8476306351c524de33fd3d15c0a4425282a874f"} Oct 30 13:30:14.525785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:14.525737 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" podStartSLOduration=2.525724048 podStartE2EDuration="2.525724048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:14.518656624 +0000 UTC m=+3432.645457523" watchObservedRunningTime="2026-10-30 13:30:14.525724048 +0000 UTC m=+3432.652524944" Oct 30 13:30:15.308203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:15.308161 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:15.359327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:15.359301 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:15.360634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:15.360608 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p95xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 13:30:15.360773 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:15.360653 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 13:30:16.028504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:16.028471 5515 scope.go:122] "RemoveContainer" containerID="45f64a57aa6a52e9282fcfda30d43b7462740e28e3741ff43b3d976ad096cc04" Oct 30 13:30:16.028917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:16.028578 5515 scope.go:122] "RemoveContainer" containerID="156919cca8c42d61f9f5e57df9bae29ee53ac95fda293f50d4807a2d7671ea64" Oct 30 13:30:16.360136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:16.360090 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p95xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 13:30:16.360298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:16.360154 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 13:30:17.010293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.007795 5515 scope.go:122] "RemoveContainer" containerID="116813f42829591c9fae329a706b340e54510b0821827b5642d02fa9da9d64b0" Oct 30 13:30:17.256199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.255369 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:17.275521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.275447 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" podUID="7b42d3ee-e0ec-496e-8004-a74605480045" containerName="payload-processing" containerID="cri-o://1fb19bc0a95f5b28637404c3cee2bf0ad0f9cfbbb89e972f5ffde1de43081052" gracePeriod=25 Oct 30 13:30:17.360667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.360643 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p95xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 13:30:17.360774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.360704 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 13:30:17.436304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.436086 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" podUID="86455459-c6d6-4df9-90b6-e9cb8a718fce" containerName="payload-processing" containerID="cri-o://d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1" gracePeriod=25 Oct 30 13:30:17.490181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.490133 5515 generic.go:356] "Generic (PLEG): container finished" podID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerID="bd0840fc3e4fcd585deeeddd4cf8bf9e9f1d9492acd1d777d5e07ab287c25318" exitCode=60 Oct 30 13:30:17.490334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.490231 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" event={"ID":"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0","Type":"ContainerDied","Data":"bd0840fc3e4fcd585deeeddd4cf8bf9e9f1d9492acd1d777d5e07ab287c25318"} Oct 30 13:30:17.490334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.490305 5515 scope.go:122] "RemoveContainer" containerID="45f64a57aa6a52e9282fcfda30d43b7462740e28e3741ff43b3d976ad096cc04" Oct 30 13:30:17.490854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.490831 5515 scope.go:122] "RemoveContainer" containerID="bd0840fc3e4fcd585deeeddd4cf8bf9e9f1d9492acd1d777d5e07ab287c25318" Oct 30 13:30:17.491303 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:17.491250 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-zkrq7_odh-ai-gateway-infra(b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" Oct 30 13:30:17.527044 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.525685 5515 generic.go:356] "Generic (PLEG): container finished" podID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerID="5011b84a8a185eaab15c72e6711db46cfe1194e1db23221519b108d0ddcf6ea5" exitCode=60 Oct 30 13:30:17.527044 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.525759 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" event={"ID":"52db1b88-ea91-4642-93e4-40043dddd4fc","Type":"ContainerDied","Data":"5011b84a8a185eaab15c72e6711db46cfe1194e1db23221519b108d0ddcf6ea5"} Oct 30 13:30:17.527044 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.526234 5515 scope.go:122] "RemoveContainer" containerID="5011b84a8a185eaab15c72e6711db46cfe1194e1db23221519b108d0ddcf6ea5" Oct 30 13:30:17.527044 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:17.526661 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89_odh-ai-gateway-infra(52db1b88-ea91-4642-93e4-40043dddd4fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" Oct 30 13:30:17.528788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.528720 5515 generic.go:356] "Generic (PLEG): container finished" podID="7b42d3ee-e0ec-496e-8004-a74605480045" containerID="1fb19bc0a95f5b28637404c3cee2bf0ad0f9cfbbb89e972f5ffde1de43081052" exitCode=0 Oct 30 13:30:17.528900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.528797 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" event={"ID":"7b42d3ee-e0ec-496e-8004-a74605480045","Type":"ContainerDied","Data":"1fb19bc0a95f5b28637404c3cee2bf0ad0f9cfbbb89e972f5ffde1de43081052"} Oct 30 13:30:17.554783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.554759 5515 scope.go:122] "RemoveContainer" containerID="156919cca8c42d61f9f5e57df9bae29ee53ac95fda293f50d4807a2d7671ea64" Oct 30 13:30:17.591010 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.590985 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:30:17.666296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.665717 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-routing-overlay-volume\") pod \"7b42d3ee-e0ec-496e-8004-a74605480045\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " Oct 30 13:30:17.666296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.665885 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-kube-api-access-wdnfw\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-kube-api-access-wdnfw\") pod \"7b42d3ee-e0ec-496e-8004-a74605480045\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " Oct 30 13:30:17.666296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.665959 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-provider-credentials\") pod \"7b42d3ee-e0ec-496e-8004-a74605480045\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " Oct 30 13:30:17.666296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.666041 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-plugins-config-volume\") pod \"7b42d3ee-e0ec-496e-8004-a74605480045\" (UID: \"7b42d3ee-e0ec-496e-8004-a74605480045\") " Oct 30 13:30:17.667232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.667204 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-routing-overlay-volume" pod "7b42d3ee-e0ec-496e-8004-a74605480045" (UID: "7b42d3ee-e0ec-496e-8004-a74605480045"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:17.667719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.667701 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-plugins-config-volume" pod "7b42d3ee-e0ec-496e-8004-a74605480045" (UID: "7b42d3ee-e0ec-496e-8004-a74605480045"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:17.670217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.670104 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-provider-credentials" pod "7b42d3ee-e0ec-496e-8004-a74605480045" (UID: "7b42d3ee-e0ec-496e-8004-a74605480045"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.670337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.670255 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-kube-api-access-wdnfw" pod "7b42d3ee-e0ec-496e-8004-a74605480045" (UID: "7b42d3ee-e0ec-496e-8004-a74605480045"). InnerVolumeSpecName "kube-api-access-wdnfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.744909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.744887 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:17.768454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.768424 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.768454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.768451 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdnfw\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-kube-api-access-wdnfw\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.768633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.768462 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b42d3ee-e0ec-496e-8004-a74605480045-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.768633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.768472 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b42d3ee-e0ec-496e-8004-a74605480045-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.869384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.869296 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-plugins-config-volume\") pod \"86455459-c6d6-4df9-90b6-e9cb8a718fce\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " Oct 30 13:30:17.869549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.869388 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-kube-api-access-p6lgq\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-kube-api-access-p6lgq\") pod \"86455459-c6d6-4df9-90b6-e9cb8a718fce\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " Oct 30 13:30:17.869549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.869409 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-provider-credentials\") pod \"86455459-c6d6-4df9-90b6-e9cb8a718fce\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " Oct 30 13:30:17.869549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.869463 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-routing-overlay-volume\") pod \"86455459-c6d6-4df9-90b6-e9cb8a718fce\" (UID: \"86455459-c6d6-4df9-90b6-e9cb8a718fce\") " Oct 30 13:30:17.869721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.869651 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-plugins-config-volume" pod "86455459-c6d6-4df9-90b6-e9cb8a718fce" (UID: "86455459-c6d6-4df9-90b6-e9cb8a718fce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:17.869910 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.869878 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-routing-overlay-volume" pod "86455459-c6d6-4df9-90b6-e9cb8a718fce" (UID: "86455459-c6d6-4df9-90b6-e9cb8a718fce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:17.870127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.870109 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.870189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.870132 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86455459-c6d6-4df9-90b6-e9cb8a718fce-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.871483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.871461 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-kube-api-access-p6lgq" pod "86455459-c6d6-4df9-90b6-e9cb8a718fce" (UID: "86455459-c6d6-4df9-90b6-e9cb8a718fce"). InnerVolumeSpecName "kube-api-access-p6lgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.871835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.871812 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-provider-credentials" pod "86455459-c6d6-4df9-90b6-e9cb8a718fce" (UID: "86455459-c6d6-4df9-90b6-e9cb8a718fce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.971847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.971806 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6lgq\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-kube-api-access-p6lgq\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.971847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:17.971839 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86455459-c6d6-4df9-90b6-e9cb8a718fce-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.019293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.018879 5515 scope.go:122] "RemoveContainer" containerID="a9a94f14e62b42ce4a0beb11e50ea33f2898a2febe8ace38906ec30179f58856" Oct 30 13:30:18.359803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.359776 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p95xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 13:30:18.360167 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.359832 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 13:30:18.538059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.538033 5515 generic.go:356] "Generic (PLEG): container finished" podID="86455459-c6d6-4df9-90b6-e9cb8a718fce" containerID="d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1" exitCode=0 Oct 30 13:30:18.538181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.538071 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" Oct 30 13:30:18.538181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.538108 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" event={"ID":"86455459-c6d6-4df9-90b6-e9cb8a718fce","Type":"ContainerDied","Data":"d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1"} Oct 30 13:30:18.538181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.538162 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd" event={"ID":"86455459-c6d6-4df9-90b6-e9cb8a718fce","Type":"ContainerDied","Data":"3ee5cbfe418334faf8366af75d7bea9385f0d2ca6fa1b49a5e6fc011d7476c4a"} Oct 30 13:30:18.538309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.538187 5515 scope.go:122] "RemoveContainer" containerID="d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1" Oct 30 13:30:18.543791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.543771 5515 generic.go:356] "Generic (PLEG): container finished" podID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerID="af888a3cb4549e3471dcfd67d6998911f890326fef537f63cde830094b42f229" exitCode=60 Oct 30 13:30:18.543908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.543839 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" event={"ID":"c92ce883-ec8c-47e9-b269-019f5d507d75","Type":"ContainerDied","Data":"af888a3cb4549e3471dcfd67d6998911f890326fef537f63cde830094b42f229"} Oct 30 13:30:18.544367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.544352 5515 scope.go:122] "RemoveContainer" containerID="af888a3cb4549e3471dcfd67d6998911f890326fef537f63cde830094b42f229" Oct 30 13:30:18.544703 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:18.544684 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7_odh-ai-gateway-infra(c92ce883-ec8c-47e9-b269-019f5d507d75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" Oct 30 13:30:18.545394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.545380 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" Oct 30 13:30:18.545478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.545429 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7" event={"ID":"7b42d3ee-e0ec-496e-8004-a74605480045","Type":"ContainerDied","Data":"6fac073cd4dc2c360d1a6c193ee4c67b7ed53393873bbe9647e5ccf6b68803a1"} Oct 30 13:30:18.547716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.547697 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" event={"ID":"7522726f-7b19-45ef-92eb-802bcb81be87","Type":"ContainerStarted","Data":"edbb97a9549177fec2f7a2cdf96a008b333ba726e9b564a499bc8e2b96d73e66"} Oct 30 13:30:18.549184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.549167 5515 scope.go:122] "RemoveContainer" containerID="d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1" Oct 30 13:30:18.549450 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:18.549434 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1\": container with ID starting with d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1 not found: ID does not exist" containerID="d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1" Oct 30 13:30:18.549509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.549456 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1"} err="failed to get container status \"d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1\": rpc error: code = NotFound desc = could not find container \"d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1\": container with ID starting with d518717dd3632f9d96a41948aa2d2f47da2ee99927df9ce0922f3658383c8ee1 not found: ID does not exist" Oct 30 13:30:18.549509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.549470 5515 scope.go:122] "RemoveContainer" containerID="116813f42829591c9fae329a706b340e54510b0821827b5642d02fa9da9d64b0" Oct 30 13:30:18.563578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.563493 5515 scope.go:122] "RemoveContainer" containerID="1fb19bc0a95f5b28637404c3cee2bf0ad0f9cfbbb89e972f5ffde1de43081052" Oct 30 13:30:18.569580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.569562 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7"] Oct 30 13:30:18.633161 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.633127 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89"] Oct 30 13:30:18.730709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.730676 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd"] Oct 30 13:30:18.734986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.734959 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-5fb7zl5vd"] Oct 30 13:30:18.775665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.775600 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7"] Oct 30 13:30:18.782151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.782128 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-65bbbwjl7"] Oct 30 13:30:18.824356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.824167 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:18.876198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.876171 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:18.884600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.884553 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-kube-api-access-lrjq6\" (UniqueName: \"kubernetes.io/projected/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-kube-api-access-lrjq6\") pod \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\" (UID: \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\") " Oct 30 13:30:18.884600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.884587 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-maas-api-service-ca\") pod \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\" (UID: \"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0\") " Oct 30 13:30:18.884944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.884919 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-maas-api-service-ca" pod "b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" (UID: "b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:18.885255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.885239 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.886726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.886707 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-kube-api-access-lrjq6" pod "b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" (UID: "b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0"). InnerVolumeSpecName "kube-api-access-lrjq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:18.985985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.985954 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52db1b88-ea91-4642-93e4-40043dddd4fc-kube-api-access-jms4m\" (UniqueName: \"kubernetes.io/projected/52db1b88-ea91-4642-93e4-40043dddd4fc-kube-api-access-jms4m\") pod \"52db1b88-ea91-4642-93e4-40043dddd4fc\" (UID: \"52db1b88-ea91-4642-93e4-40043dddd4fc\") " Oct 30 13:30:18.986189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.986155 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52db1b88-ea91-4642-93e4-40043dddd4fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52db1b88-ea91-4642-93e4-40043dddd4fc-maas-api-service-ca\") pod \"52db1b88-ea91-4642-93e4-40043dddd4fc\" (UID: \"52db1b88-ea91-4642-93e4-40043dddd4fc\") " Oct 30 13:30:18.986754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.986727 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrjq6\" (UniqueName: \"kubernetes.io/projected/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0-kube-api-access-lrjq6\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.986754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.986741 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52db1b88-ea91-4642-93e4-40043dddd4fc-maas-api-service-ca" pod "52db1b88-ea91-4642-93e4-40043dddd4fc" (UID: "52db1b88-ea91-4642-93e4-40043dddd4fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:18.988561 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:18.988535 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52db1b88-ea91-4642-93e4-40043dddd4fc-kube-api-access-jms4m" pod "52db1b88-ea91-4642-93e4-40043dddd4fc" (UID: "52db1b88-ea91-4642-93e4-40043dddd4fc"). InnerVolumeSpecName "kube-api-access-jms4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:19.019344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.019161 5515 scope.go:122] "RemoveContainer" containerID="cab58928c32925646b135a18bef9388c263bb030987d3076d72805b9cdb8ba35" Oct 30 13:30:19.089897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.088710 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52db1b88-ea91-4642-93e4-40043dddd4fc-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.089897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.088749 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jms4m\" (UniqueName: \"kubernetes.io/projected/52db1b88-ea91-4642-93e4-40043dddd4fc-kube-api-access-jms4m\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.144994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.144964 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:19.360412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.360338 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p95xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 13:30:19.360412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.360387 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 13:30:19.555977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.555943 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" event={"ID":"b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0","Type":"ContainerDied","Data":"cc799f6fd6e5e6dd12bfdcbc11126bedecfcba6ee1e2b00e799f7909552a83a5"} Oct 30 13:30:19.556150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.556014 5515 scope.go:122] "RemoveContainer" containerID="bd0840fc3e4fcd585deeeddd4cf8bf9e9f1d9492acd1d777d5e07ab287c25318" Oct 30 13:30:19.556150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.555953 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7" Oct 30 13:30:19.557595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.557580 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" Oct 30 13:30:19.557700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.557601 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89" event={"ID":"52db1b88-ea91-4642-93e4-40043dddd4fc","Type":"ContainerDied","Data":"5934a424f742b4f56c84249dcd51f9d13aa793e0411b2eeddb2109bfcb40a2cb"} Oct 30 13:30:19.566525 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.566505 5515 generic.go:356] "Generic (PLEG): container finished" podID="7522726f-7b19-45ef-92eb-802bcb81be87" containerID="edbb97a9549177fec2f7a2cdf96a008b333ba726e9b564a499bc8e2b96d73e66" exitCode=60 Oct 30 13:30:19.566604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.566574 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" event={"ID":"7522726f-7b19-45ef-92eb-802bcb81be87","Type":"ContainerDied","Data":"edbb97a9549177fec2f7a2cdf96a008b333ba726e9b564a499bc8e2b96d73e66"} Oct 30 13:30:19.566980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.566963 5515 scope.go:122] "RemoveContainer" containerID="edbb97a9549177fec2f7a2cdf96a008b333ba726e9b564a499bc8e2b96d73e66" Oct 30 13:30:19.567261 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:19.567241 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w_odh-ai-gateway-infra(7522726f-7b19-45ef-92eb-802bcb81be87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" Oct 30 13:30:19.568355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.568155 5515 scope.go:122] "RemoveContainer" containerID="5011b84a8a185eaab15c72e6711db46cfe1194e1db23221519b108d0ddcf6ea5" Oct 30 13:30:19.569901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.569882 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" event={"ID":"aceff121-3e3e-4298-b667-e0b14aade56d","Type":"ContainerStarted","Data":"b5b6fb04b5bca9e307baaa1f0a9d292ef71333d4b008805b7e5670608fa73439"} Oct 30 13:30:19.578125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.578110 5515 scope.go:122] "RemoveContainer" containerID="a9a94f14e62b42ce4a0beb11e50ea33f2898a2febe8ace38906ec30179f58856" Oct 30 13:30:19.617372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.617285 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7"] Oct 30 13:30:19.620835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.620806 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-zkrq7"] Oct 30 13:30:19.639323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.639294 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w"] Oct 30 13:30:19.661967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.661936 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89"] Oct 30 13:30:19.669510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.669480 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-241ffa-29889450-hlk89"] Oct 30 13:30:19.687536 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.687507 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7"] Oct 30 13:30:19.883173 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:19.883143 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:20.006226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.006187 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" path="/var/lib/kubelet/pods/52db1b88-ea91-4642-93e4-40043dddd4fc/volumes" Oct 30 13:30:20.006791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.006770 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b42d3ee-e0ec-496e-8004-a74605480045" path="/var/lib/kubelet/pods/7b42d3ee-e0ec-496e-8004-a74605480045/volumes" Oct 30 13:30:20.007243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.007224 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86455459-c6d6-4df9-90b6-e9cb8a718fce" path="/var/lib/kubelet/pods/86455459-c6d6-4df9-90b6-e9cb8a718fce/volumes" Oct 30 13:30:20.008005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.007985 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" path="/var/lib/kubelet/pods/b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0/volumes" Oct 30 13:30:20.015668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.015645 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92ce883-ec8c-47e9-b269-019f5d507d75-kube-api-access-9slpc\" (UniqueName: \"kubernetes.io/projected/c92ce883-ec8c-47e9-b269-019f5d507d75-kube-api-access-9slpc\") pod \"c92ce883-ec8c-47e9-b269-019f5d507d75\" (UID: \"c92ce883-ec8c-47e9-b269-019f5d507d75\") " Oct 30 13:30:20.015774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.015680 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92ce883-ec8c-47e9-b269-019f5d507d75-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92ce883-ec8c-47e9-b269-019f5d507d75-maas-api-service-ca\") pod \"c92ce883-ec8c-47e9-b269-019f5d507d75\" (UID: \"c92ce883-ec8c-47e9-b269-019f5d507d75\") " Oct 30 13:30:20.016094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.016069 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92ce883-ec8c-47e9-b269-019f5d507d75-maas-api-service-ca" pod "c92ce883-ec8c-47e9-b269-019f5d507d75" (UID: "c92ce883-ec8c-47e9-b269-019f5d507d75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:20.016224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.016209 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c92ce883-ec8c-47e9-b269-019f5d507d75-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.017842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.017816 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92ce883-ec8c-47e9-b269-019f5d507d75-kube-api-access-9slpc" pod "c92ce883-ec8c-47e9-b269-019f5d507d75" (UID: "c92ce883-ec8c-47e9-b269-019f5d507d75"). InnerVolumeSpecName "kube-api-access-9slpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:20.117581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.117545 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9slpc\" (UniqueName: \"kubernetes.io/projected/c92ce883-ec8c-47e9-b269-019f5d507d75-kube-api-access-9slpc\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.359754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.359721 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p95xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 13:30:20.359966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.359939 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 13:30:20.527611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.527534 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:20.582696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.582670 5515 generic.go:356] "Generic (PLEG): container finished" podID="aceff121-3e3e-4298-b667-e0b14aade56d" containerID="b5b6fb04b5bca9e307baaa1f0a9d292ef71333d4b008805b7e5670608fa73439" exitCode=60 Oct 30 13:30:20.582796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.582739 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" event={"ID":"aceff121-3e3e-4298-b667-e0b14aade56d","Type":"ContainerDied","Data":"b5b6fb04b5bca9e307baaa1f0a9d292ef71333d4b008805b7e5670608fa73439"} Oct 30 13:30:20.582875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.582805 5515 scope.go:122] "RemoveContainer" containerID="cab58928c32925646b135a18bef9388c263bb030987d3076d72805b9cdb8ba35" Oct 30 13:30:20.583314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.583242 5515 scope.go:122] "RemoveContainer" containerID="b5b6fb04b5bca9e307baaa1f0a9d292ef71333d4b008805b7e5670608fa73439" Oct 30 13:30:20.583737 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:20.583709 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h_odh-ai-gateway-infra(aceff121-3e3e-4298-b667-e0b14aade56d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" Oct 30 13:30:20.586734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.586719 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" Oct 30 13:30:20.586813 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.586764 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7" event={"ID":"c92ce883-ec8c-47e9-b269-019f5d507d75","Type":"ContainerDied","Data":"f0803aba186eee5149c5608dbc882c6a9ea8af86c54b7d890a73a96f15068dbb"} Oct 30 13:30:20.598554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.598532 5515 scope.go:122] "RemoveContainer" containerID="af888a3cb4549e3471dcfd67d6998911f890326fef537f63cde830094b42f229" Oct 30 13:30:20.674288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.672362 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7"] Oct 30 13:30:20.677761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.677708 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e9cb4a-29889450-nzkd7"] Oct 30 13:30:20.759277 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.759243 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:20.775621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.775593 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h"] Oct 30 13:30:20.775878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.775836 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:20.931672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.931635 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7522726f-7b19-45ef-92eb-802bcb81be87-kube-api-access-wmn2v\" (UniqueName: \"kubernetes.io/projected/7522726f-7b19-45ef-92eb-802bcb81be87-kube-api-access-wmn2v\") pod \"7522726f-7b19-45ef-92eb-802bcb81be87\" (UID: \"7522726f-7b19-45ef-92eb-802bcb81be87\") " Oct 30 13:30:20.931863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.931798 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7522726f-7b19-45ef-92eb-802bcb81be87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7522726f-7b19-45ef-92eb-802bcb81be87-maas-api-service-ca\") pod \"7522726f-7b19-45ef-92eb-802bcb81be87\" (UID: \"7522726f-7b19-45ef-92eb-802bcb81be87\") " Oct 30 13:30:20.932228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.932189 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7522726f-7b19-45ef-92eb-802bcb81be87-maas-api-service-ca" pod "7522726f-7b19-45ef-92eb-802bcb81be87" (UID: "7522726f-7b19-45ef-92eb-802bcb81be87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:20.932396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.932380 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7522726f-7b19-45ef-92eb-802bcb81be87-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.933910 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:20.933891 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7522726f-7b19-45ef-92eb-802bcb81be87-kube-api-access-wmn2v" pod "7522726f-7b19-45ef-92eb-802bcb81be87" (UID: "7522726f-7b19-45ef-92eb-802bcb81be87"). InnerVolumeSpecName "kube-api-access-wmn2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:21.013335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.013262 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:21.033260 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.033236 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmn2v\" (UniqueName: \"kubernetes.io/projected/7522726f-7b19-45ef-92eb-802bcb81be87-kube-api-access-wmn2v\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.360124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.359986 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p95xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 13:30:21.360124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.360044 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 13:30:21.429673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.429638 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:30:21.607288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.607256 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" Oct 30 13:30:21.607681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.607257 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w" event={"ID":"7522726f-7b19-45ef-92eb-802bcb81be87","Type":"ContainerDied","Data":"b1d47a9df11a484b1b80bd3c7eee89018121a00cadd7f517de4e64c5b340fbb1"} Oct 30 13:30:21.607681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.607371 5515 scope.go:122] "RemoveContainer" containerID="edbb97a9549177fec2f7a2cdf96a008b333ba726e9b564a499bc8e2b96d73e66" Oct 30 13:30:21.708184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.707660 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w"] Oct 30 13:30:21.723932 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.713782 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889450-99d4w"] Oct 30 13:30:21.836939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.836914 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:21.944920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.944884 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aceff121-3e3e-4298-b667-e0b14aade56d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aceff121-3e3e-4298-b667-e0b14aade56d-maas-api-service-ca\") pod \"aceff121-3e3e-4298-b667-e0b14aade56d\" (UID: \"aceff121-3e3e-4298-b667-e0b14aade56d\") " Oct 30 13:30:21.945104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.945022 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aceff121-3e3e-4298-b667-e0b14aade56d-kube-api-access-x6pfc\" (UniqueName: \"kubernetes.io/projected/aceff121-3e3e-4298-b667-e0b14aade56d-kube-api-access-x6pfc\") pod \"aceff121-3e3e-4298-b667-e0b14aade56d\" (UID: \"aceff121-3e3e-4298-b667-e0b14aade56d\") " Oct 30 13:30:21.945406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.945378 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aceff121-3e3e-4298-b667-e0b14aade56d-maas-api-service-ca" pod "aceff121-3e3e-4298-b667-e0b14aade56d" (UID: "aceff121-3e3e-4298-b667-e0b14aade56d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:21.945731 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.945707 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aceff121-3e3e-4298-b667-e0b14aade56d-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.948187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:21.948157 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aceff121-3e3e-4298-b667-e0b14aade56d-kube-api-access-x6pfc" pod "aceff121-3e3e-4298-b667-e0b14aade56d" (UID: "aceff121-3e3e-4298-b667-e0b14aade56d"). InnerVolumeSpecName "kube-api-access-x6pfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:22.006244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.006211 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" path="/var/lib/kubelet/pods/7522726f-7b19-45ef-92eb-802bcb81be87/volumes" Oct 30 13:30:22.006747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.006730 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" path="/var/lib/kubelet/pods/c92ce883-ec8c-47e9-b269-019f5d507d75/volumes" Oct 30 13:30:22.047025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.046986 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6pfc\" (UniqueName: \"kubernetes.io/projected/aceff121-3e3e-4298-b667-e0b14aade56d-kube-api-access-x6pfc\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:30:22.367202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.367115 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:22.367476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.367456 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:22.368522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.368499 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:30:22.631775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.631689 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" Oct 30 13:30:22.632261 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.631867 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h" event={"ID":"aceff121-3e3e-4298-b667-e0b14aade56d","Type":"ContainerDied","Data":"ef2a8e4b24e9088f4e549feb390aa4a4d19f1302da9e5e5dd1fdf5911aa99838"} Oct 30 13:30:22.632261 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.631955 5515 scope.go:122] "RemoveContainer" containerID="b5b6fb04b5bca9e307baaa1f0a9d292ef71333d4b008805b7e5670608fa73439" Oct 30 13:30:22.697951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.697918 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h"] Oct 30 13:30:22.702779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.702703 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889450-f4t4h"] Oct 30 13:30:22.963350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:22.963244 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:24.004385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:24.004357 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" path="/var/lib/kubelet/pods/aceff121-3e3e-4298-b667-e0b14aade56d/volumes" Oct 30 13:30:24.768451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:24.768421 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:26.475523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:26.475490 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:27.410468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:27.410435 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:27.612830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:27.612791 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:27.914310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:27.914260 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:28.222595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:28.222494 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:28.437453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:28.437415 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:28.952305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:28.950902 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:28.995061 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:28.995026 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:30:29.181457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:29.181383 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:30.362782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:30.362749 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:30.571279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:30.571233 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:31.260018 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.259968 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:31.404094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404054 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5"] Oct 30 13:30:31.404891 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404875 5515 cpu_manager.go:414] "containerMap: removing container" podUID="86455459-c6d6-4df9-90b6-e9cb8a718fce" containerName="payload-processing" Oct 30 13:30:31.404937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404892 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86455459-c6d6-4df9-90b6-e9cb8a718fce" containerName="payload-processing" Oct 30 13:30:31.404937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404910 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerName="cleanup" Oct 30 13:30:31.404937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404915 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerName="cleanup" Oct 30 13:30:31.404937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404922 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerName="cleanup" Oct 30 13:30:31.404937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404928 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerName="cleanup" Oct 30 13:30:31.404937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404938 5515 cpu_manager.go:414] "containerMap: removing container" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404942 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404952 5515 cpu_manager.go:414] "containerMap: removing container" podUID="7b42d3ee-e0ec-496e-8004-a74605480045" containerName="payload-processing" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404956 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b42d3ee-e0ec-496e-8004-a74605480045" containerName="payload-processing" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404963 5515 cpu_manager.go:414] "containerMap: removing container" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404967 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404974 5515 cpu_manager.go:414] "containerMap: removing container" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404979 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404992 5515 cpu_manager.go:414] "containerMap: removing container" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.404996 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405003 5515 cpu_manager.go:414] "containerMap: removing container" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405007 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405012 5515 cpu_manager.go:414] "containerMap: removing container" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405016 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405023 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405028 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405034 5515 cpu_manager.go:414] "containerMap: removing container" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405038 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405044 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405048 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405063 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerName="cleanup" Oct 30 13:30:31.405130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405068 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405164 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="86455459-c6d6-4df9-90b6-e9cb8a718fce" containerName="payload-processing" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405174 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405180 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405186 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405194 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b42d3ee-e0ec-496e-8004-a74605480045" containerName="payload-processing" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405200 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405206 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405215 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405219 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405225 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405230 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405238 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerName="cleanup" Oct 30 13:30:31.405859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.405243 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerName="cleanup" Oct 30 13:30:31.410449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.410429 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.431217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.431193 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5"] Oct 30 13:30:31.439521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.439495 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"agc-cred-7337afd955bc\"" Oct 30 13:30:31.439702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.439512 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"kube-root-ca.crt\"" Oct 30 13:30:31.439702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.439560 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"openshift-service-ca.crt\"" Oct 30 13:30:31.440366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.440324 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-acf9e2\"" Oct 30 13:30:31.440366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.440341 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"payload-processing-external-model-e2e--66ed339f-dockercfg-sh52l\"" Oct 30 13:30:31.440945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.440927 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"routing-overlay\"" Oct 30 13:30:31.524336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.517251 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.524336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.517415 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.524336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.517464 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.524336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.517624 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvg7\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-kube-api-access-7dvg7\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.619486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.619448 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvg7\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-kube-api-access-7dvg7\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.619672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.619603 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.619716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.619702 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.619763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.619748 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.620943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.620490 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.621102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.620942 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.624077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.624013 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.645192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.645164 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvg7\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-kube-api-access-7dvg7\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:31.726003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:31.725969 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:32.032497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:32.032472 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5"] Oct 30 13:30:32.033189 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:30:32.033162 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689fb278_7e18_491e_94c2_2e37cf52a896.slice/crio-7ef6818fb4ec64949ead0def7a0f9fbd5d1d6b782829387c445f27e4c351819a WatchSource:0}: Error finding container 7ef6818fb4ec64949ead0def7a0f9fbd5d1d6b782829387c445f27e4c351819a: Status 404 returned error can't find the container with id 7ef6818fb4ec64949ead0def7a0f9fbd5d1d6b782829387c445f27e4c351819a Oct 30 13:30:32.311745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:32.311636 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:32.733131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:32.733094 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" event={"ID":"689fb278-7e18-491e-94c2-2e37cf52a896","Type":"ContainerStarted","Data":"4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a"} Oct 30 13:30:32.733581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:32.733150 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" event={"ID":"689fb278-7e18-491e-94c2-2e37cf52a896","Type":"ContainerStarted","Data":"7ef6818fb4ec64949ead0def7a0f9fbd5d1d6b782829387c445f27e4c351819a"} Oct 30 13:30:32.733581 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:32.733293 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:32.777224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:32.777176 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" podStartSLOduration=1.777163257 podStartE2EDuration="1.777163257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:32.776158428 +0000 UTC m=+3450.902959327" watchObservedRunningTime="2026-10-30 13:30:32.777163257 +0000 UTC m=+3450.903964154" Oct 30 13:30:34.049005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:34.048954 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:35.890663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:35.890628 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:36.293642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:36.293549 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:30:38.005347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:38.003751 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:39.814307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:39.812918 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:41.581005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:41.580971 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:43.219614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:43.219572 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:43.743892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:43.743858 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:30:44.926535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:44.926500 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:46.763894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:46.763859 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:48.622674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:48.622643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:50.319630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:50.319597 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:52.022068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:52.022037 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:52.509987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:52.509951 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:52.726708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:52.726673 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:52.943065 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:52.943006 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:53.777937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:53.777879 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:54.005097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:54.005067 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:54.366956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:54.366924 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:54.577291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:54.576805 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:55.713681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:55.713646 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:57.183256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:57.183222 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:57.390829 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:57.390799 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:57.600019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:57.599939 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:30:58.989537 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:30:58.989501 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:30:59.367835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:30:59.367760 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:01.019856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:01.019821 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:02.161713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:02.161679 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:02.375319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:02.375287 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:02.952783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:02.952750 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:04.608767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:04.608736 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:06.414570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:06.414534 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:08.209712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:08.209676 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:09.105642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:09.105613 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:09.348776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:09.348741 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:09.573943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:09.573905 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:09.881099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:09.881024 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:10.096610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:10.096577 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:10.551508 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:10.551479 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:10.759752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:10.759713 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:11.876252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:11.876208 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:12.724983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:12.724950 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:13.130951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.130877 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:13.174801 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.174766 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv"] Oct 30 13:31:13.175704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175675 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerName="cleanup" Oct 30 13:31:13.175704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175701 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerName="cleanup" Oct 30 13:31:13.175893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175734 5515 cpu_manager.go:414] "containerMap: removing container" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" containerName="cleanup" Oct 30 13:31:13.175893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175742 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" containerName="cleanup" Oct 30 13:31:13.175893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175770 5515 cpu_manager.go:414] "containerMap: removing container" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerName="cleanup" Oct 30 13:31:13.175893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175777 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52db1b88-ea91-4642-93e4-40043dddd4fc" containerName="cleanup" Oct 30 13:31:13.176039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175947 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92ce883-ec8c-47e9-b269-019f5d507d75" containerName="cleanup" Oct 30 13:31:13.176039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175962 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ddb74d-4ed2-4c29-bf7b-6636eaa659f0" containerName="cleanup" Oct 30 13:31:13.176039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175972 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aceff121-3e3e-4298-b667-e0b14aade56d" containerName="cleanup" Oct 30 13:31:13.176039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.175981 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7522726f-7b19-45ef-92eb-802bcb81be87" containerName="cleanup" Oct 30 13:31:13.179577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.179557 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.210071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.210028 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcmth\" (UniqueName: \"kubernetes.io/projected/e8370493-9181-48ef-bd2e-b3eab3293f98-kube-api-access-wcmth\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.210386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.210220 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.210386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.210293 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.210386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.210333 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.210596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.210406 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8370493-9181-48ef-bd2e-b3eab3293f98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.210596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.210523 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.210596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.210555 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.210770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.210610 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.210770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.210669 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.211776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.211754 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv"] Oct 30 13:31:13.311950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.311909 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcmth\" (UniqueName: \"kubernetes.io/projected/e8370493-9181-48ef-bd2e-b3eab3293f98-kube-api-access-wcmth\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312031 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312076 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312131 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312165 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8370493-9181-48ef-bd2e-b3eab3293f98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312313 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312354 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312408 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312464 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312529 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312548 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312668 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312776 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.312999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.312979 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8370493-9181-48ef-bd2e-b3eab3293f98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.314563 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.314545 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.314710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.314691 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.340955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.340926 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcmth\" (UniqueName: \"kubernetes.io/projected/e8370493-9181-48ef-bd2e-b3eab3293f98-kube-api-access-wcmth\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.348997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.348975 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8370493-9181-48ef-bd2e-b3eab3293f98-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-q8crv\" (UID: \"e8370493-9181-48ef-bd2e-b3eab3293f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.496751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.496723 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:13.663325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.663254 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:13.822462 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:31:13.822418 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8370493_9181_48ef_bd2e_b3eab3293f98.slice/crio-5e75938bfe4192cbab984b9da0f5ae04965b6075e5670f866cd4866d29b63fb3 WatchSource:0}: Error finding container 5e75938bfe4192cbab984b9da0f5ae04965b6075e5670f866cd4866d29b63fb3: Status 404 returned error can't find the container with id 5e75938bfe4192cbab984b9da0f5ae04965b6075e5670f866cd4866d29b63fb3 Oct 30 13:31:13.823043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.822977 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv"] Oct 30 13:31:13.825189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.825140 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:13.825335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.825216 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:13.825335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.825247 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:31:13.875200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:13.875164 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:14.097303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:14.094071 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" event={"ID":"e8370493-9181-48ef-bd2e-b3eab3293f98","Type":"ContainerStarted","Data":"16306d911131487db787d2f5ebca58458a37e5ed82773bf4abf063c093930c37"} Oct 30 13:31:14.097303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:14.094124 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" event={"ID":"e8370493-9181-48ef-bd2e-b3eab3293f98","Type":"ContainerStarted","Data":"5e75938bfe4192cbab984b9da0f5ae04965b6075e5670f866cd4866d29b63fb3"} Oct 30 13:31:14.153199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:14.153147 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" podStartSLOduration=1.153131747 podStartE2EDuration="1.153131747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:14.15046845 +0000 UTC m=+3492.277269350" watchObservedRunningTime="2026-10-30 13:31:14.153131747 +0000 UTC m=+3492.279932645" Oct 30 13:31:14.498286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:14.497715 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:14.515511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:14.514768 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-q8crv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:31:14.515511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:14.514837 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" podUID="e8370493-9181-48ef-bd2e-b3eab3293f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:31:15.503292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:15.497697 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-q8crv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:31:15.503292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:15.497760 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" podUID="e8370493-9181-48ef-bd2e-b3eab3293f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:31:15.580783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:15.580754 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:15.975591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:15.975553 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:16.183536 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:16.183501 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:16.500405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:16.497638 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-q8crv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:31:16.500405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:16.497695 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" podUID="e8370493-9181-48ef-bd2e-b3eab3293f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:31:16.618947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:16.618913 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:17.420296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:17.419893 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:17.517291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:17.517014 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-q8crv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:31:17.517291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:17.517073 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" podUID="e8370493-9181-48ef-bd2e-b3eab3293f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:31:17.644864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:17.644820 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:17.852309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:17.852207 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:18.504291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:18.497713 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-q8crv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:31:18.504291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:18.497808 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" podUID="e8370493-9181-48ef-bd2e-b3eab3293f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:31:19.223343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:19.219061 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:19.502763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:19.502669 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-q8crv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:31:19.503042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:19.503008 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" podUID="e8370493-9181-48ef-bd2e-b3eab3293f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:31:20.497872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:20.497842 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-q8crv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:31:20.498279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:20.497889 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" podUID="e8370493-9181-48ef-bd2e-b3eab3293f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:31:20.923637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:20.923598 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:21.497157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:21.497119 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-q8crv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 13:31:21.497386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:21.497180 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" podUID="e8370493-9181-48ef-bd2e-b3eab3293f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 13:31:22.728075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:22.728031 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:22.728625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:22.728587 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:22.729811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:22.729781 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-q8crv" Oct 30 13:31:22.847677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:22.847574 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:23.879222 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:23.879192 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:24.107743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:24.107713 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:24.368607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:24.368577 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:24.602707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:24.602675 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:24.853329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:24.846982 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:24.897168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:24.897134 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:31:25.461586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.461554 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs"] Oct 30 13:31:25.466518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.466502 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.493576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.493554 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"kube-root-ca.crt\"" Oct 30 13:31:25.506683 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.506661 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"routing-overlay\"" Oct 30 13:31:25.534966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.534938 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-d3093dc79fb2\"" Oct 30 13:31:25.535165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.535108 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs"] Oct 30 13:31:25.547107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.547080 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-e02df1dca653\"" Oct 30 13:31:25.547499 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.547481 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-e2e--e20aaa17-dockercfg-c4rvp\"" Oct 30 13:31:25.575402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.575372 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.575633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.575485 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.575633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.575522 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz9z7\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-kube-api-access-qz9z7\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.575633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.575575 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.592493 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.592467 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d2a0b\"" Oct 30 13:31:25.592637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.592501 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"openshift-service-ca.crt\"" Oct 30 13:31:25.677139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.677104 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.677335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.677248 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.677433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.677387 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.677433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.677414 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz9z7\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-kube-api-access-qz9z7\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.677880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.677854 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.678175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.678148 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.679992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.679969 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.703878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.703856 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz9z7\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-kube-api-access-qz9z7\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:25.782449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:25.782368 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:26.138901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.138868 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs"] Oct 30 13:31:26.255968 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.255936 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" event={"ID":"755120a9-0431-406c-8f80-1f24374564cb","Type":"ContainerStarted","Data":"e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c"} Oct 30 13:31:26.256180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.255994 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" event={"ID":"755120a9-0431-406c-8f80-1f24374564cb","Type":"ContainerStarted","Data":"9ed8398abd03a9e274a410ca0b3683509c3e8c1fbec83734e187fd645787467f"} Oct 30 13:31:26.256180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.256024 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:26.302882 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.302828 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" podStartSLOduration=1.302814198 podStartE2EDuration="1.302814198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:26.299568137 +0000 UTC m=+3504.426369033" watchObservedRunningTime="2026-10-30 13:31:26.302814198 +0000 UTC m=+3504.429615095" Oct 30 13:31:26.528125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.528067 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v"] Oct 30 13:31:26.569919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.537616 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.569919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.568996 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"agc-cred-adafb933cb9f\"" Oct 30 13:31:26.569919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.569094 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"agc-cred-680031a76aa7\"" Oct 30 13:31:26.569919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.569299 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-e2e--8217e17a-dockercfg-v72th\"" Oct 30 13:31:26.569919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.569377 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"kube-root-ca.crt\"" Oct 30 13:31:26.569919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.569435 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"openshift-service-ca.crt\"" Oct 30 13:31:26.569919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.569502 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1957dc\"" Oct 30 13:31:26.569919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.569539 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"routing-overlay\"" Oct 30 13:31:26.598642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.598612 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v"] Oct 30 13:31:26.604373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.602183 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.604373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.602239 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.604373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.602369 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7bhw\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-kube-api-access-d7bhw\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.604373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.602481 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.709289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.706196 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7bhw\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-kube-api-access-d7bhw\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.709289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.706351 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.709289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.706634 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.709289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.706680 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.719927 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.718014 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.719927 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.718253 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.741355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.736898 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7bhw\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-kube-api-access-d7bhw\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.755352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.754570 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:26.865777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.865686 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:26.898128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:26.898096 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:27.248675 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:31:27.248640 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42d2f44_6c24_4dc6_bf69_6f2049f34a0c.slice/crio-6d0444eeda65839ae359383151f481f8f14d288e15bf3f61e5d9e4bcbf0cfce4 WatchSource:0}: Error finding container 6d0444eeda65839ae359383151f481f8f14d288e15bf3f61e5d9e4bcbf0cfce4: Status 404 returned error can't find the container with id 6d0444eeda65839ae359383151f481f8f14d288e15bf3f61e5d9e4bcbf0cfce4 Oct 30 13:31:27.249039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:27.248697 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v"] Oct 30 13:31:27.265443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:27.265400 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" event={"ID":"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c","Type":"ContainerStarted","Data":"6d0444eeda65839ae359383151f481f8f14d288e15bf3f61e5d9e4bcbf0cfce4"} Oct 30 13:31:28.273449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:28.273413 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" event={"ID":"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c","Type":"ContainerStarted","Data":"09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505"} Oct 30 13:31:28.273844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:28.273596 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:28.321725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:28.321677 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" podStartSLOduration=2.321664936 podStartE2EDuration="2.321664936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:28.317447277 +0000 UTC m=+3506.444248174" watchObservedRunningTime="2026-10-30 13:31:28.321664936 +0000 UTC m=+3506.448465833" Oct 30 13:31:28.861644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:28.861610 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:28.990065 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:31:28.990027 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:31:30.078516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:30.078482 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:30.304044 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:30.303507 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:30.794664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:30.794633 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:31.226331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:31.226292 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:31.453537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:31.453501 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:32.359742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:32.359710 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:32.577050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:32.577007 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:32.973754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:32.973720 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:33.391334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:33.391199 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:34.888521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:34.888478 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:36.490023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:36.489990 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:37.268974 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:37.268939 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:31:38.208000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:38.207955 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:39.283991 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:39.283965 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:31:39.922732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:39.922696 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:41.677469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:41.677437 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:43.401427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:43.401386 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:45.205516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:45.205482 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:46.820172 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:46.820136 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:48.527687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:48.527605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:50.222846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:50.222811 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:51.865330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:51.865279 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:53.520119 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:53.520056 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:55.336365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:55.336323 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:57.471585 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:57.471550 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:58.750308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:31:58.750181 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:31:58.992566 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:31:58.992530 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:32:00.554376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:00.554332 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:02.578602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:02.578559 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:04.743645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:04.743611 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:06.479802 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:06.479763 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:08.294008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:08.293968 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:10.183314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:10.183280 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:11.914970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:11.914934 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:13.605875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:13.605837 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:13.847549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:13.847513 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:15.322935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:15.322894 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:17.147091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:17.147056 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:18.021635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:18.021602 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:18.242100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:18.242064 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:19.261073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:19.261040 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:20.228922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:20.228891 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:20.828947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:20.828903 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:21.131555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:21.131470 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:21.399758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:21.399726 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:23.235159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:23.235121 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:23.535320 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:23.535222 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:32:25.574081 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:25.574049 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:25.787982 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:25.787952 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:26.106447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:26.106405 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:27.266732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:27.266697 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:27.511799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:27.511753 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:27.746704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:27.746668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:28.989572 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:32:28.989530 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:32:29.644948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:29.644904 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:31.549096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:31.549060 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:33.281950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:33.281907 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:34.923211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:34.923173 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:36.624004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:36.623965 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:38.289004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:38.288907 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:39.890467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:39.890429 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:41.537227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:41.537176 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:43.207937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:43.207903 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:44.930652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:44.930614 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:46.581340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:46.581296 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:48.450673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:48.450605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:50.709573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:50.709533 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:52.583437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:52.583409 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:53.055635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:53.055589 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:53.298193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:53.298125 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:54.599085 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:54.599051 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:56.344601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:56.344514 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:57.635968 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:57.635929 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:58.097931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:58.097793 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:32:58.990134 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:32:58.990095 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:32:59.817215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:32:59.817184 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:01.587131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:01.587091 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:03.297015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:03.296981 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:03.660735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:03.660705 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:03.959553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:03.959459 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:05.247894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:05.247830 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:07.010834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:07.010804 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:07.359680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:07.359600 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp"] Oct 30 13:33:08.752593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:08.752563 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:10.471672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:10.471631 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:12.179511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.179440 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:12.360211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.360173 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" podUID="932ca447-138e-4e3d-80dd-786e0a6cf70d" containerName="payload-processing" containerID="cri-o://5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315" gracePeriod=25 Oct 30 13:33:12.444829 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.444721 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:12.673517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.673482 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:33:12.827302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.827186 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-provider-credentials\") pod \"932ca447-138e-4e3d-80dd-786e0a6cf70d\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " Oct 30 13:33:12.827506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.827260 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-kube-api-access-db6br\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-kube-api-access-db6br\") pod \"932ca447-138e-4e3d-80dd-786e0a6cf70d\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " Oct 30 13:33:12.827506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.827431 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-plugins-config-volume\") pod \"932ca447-138e-4e3d-80dd-786e0a6cf70d\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " Oct 30 13:33:12.827506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.827486 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-routing-overlay-volume\") pod \"932ca447-138e-4e3d-80dd-786e0a6cf70d\" (UID: \"932ca447-138e-4e3d-80dd-786e0a6cf70d\") " Oct 30 13:33:12.827976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.827934 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-routing-overlay-volume" pod "932ca447-138e-4e3d-80dd-786e0a6cf70d" (UID: "932ca447-138e-4e3d-80dd-786e0a6cf70d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:12.828113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.828075 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-plugins-config-volume" pod "932ca447-138e-4e3d-80dd-786e0a6cf70d" (UID: "932ca447-138e-4e3d-80dd-786e0a6cf70d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:12.828337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.828321 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:12.828403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.828340 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/932ca447-138e-4e3d-80dd-786e0a6cf70d-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:12.830130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.830097 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-provider-credentials" pod "932ca447-138e-4e3d-80dd-786e0a6cf70d" (UID: "932ca447-138e-4e3d-80dd-786e0a6cf70d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:12.830233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.830143 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-kube-api-access-db6br" pod "932ca447-138e-4e3d-80dd-786e0a6cf70d" (UID: "932ca447-138e-4e3d-80dd-786e0a6cf70d"). InnerVolumeSpecName "kube-api-access-db6br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:12.929868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.929832 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:12.929868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:12.929871 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db6br\" (UniqueName: \"kubernetes.io/projected/932ca447-138e-4e3d-80dd-786e0a6cf70d-kube-api-access-db6br\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:13.287535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:13.287504 5515 generic.go:356] "Generic (PLEG): container finished" podID="932ca447-138e-4e3d-80dd-786e0a6cf70d" containerID="5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315" exitCode=0 Oct 30 13:33:13.287954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:13.287550 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" Oct 30 13:33:13.287954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:13.287581 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" event={"ID":"932ca447-138e-4e3d-80dd-786e0a6cf70d","Type":"ContainerDied","Data":"5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315"} Oct 30 13:33:13.287954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:13.287633 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp" event={"ID":"932ca447-138e-4e3d-80dd-786e0a6cf70d","Type":"ContainerDied","Data":"47ccc0b088bd1e23edb4a9740f9939923841856554d7f983fe2cc8417b741f6e"} Oct 30 13:33:13.287954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:13.287651 5515 scope.go:122] "RemoveContainer" containerID="5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315" Oct 30 13:33:13.307666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:13.307644 5515 scope.go:122] "RemoveContainer" containerID="5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315" Oct 30 13:33:13.308001 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:33:13.307976 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315\": container with ID starting with 5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315 not found: ID does not exist" containerID="5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315" Oct 30 13:33:13.308174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:13.308012 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315"} err="failed to get container status \"5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315\": rpc error: code = NotFound desc = could not find container \"5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315\": container with ID starting with 5e02e005219abcb2be55761a590b8b830bfff8282456604f1d62c187e58ca315 not found: ID does not exist" Oct 30 13:33:13.352668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:13.352639 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp"] Oct 30 13:33:13.363057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:13.363031 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848cxltsp"] Oct 30 13:33:14.005280 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:14.005233 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932ca447-138e-4e3d-80dd-786e0a6cf70d" path="/var/lib/kubelet/pods/932ca447-138e-4e3d-80dd-786e0a6cf70d/volumes" Oct 30 13:33:14.035514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:14.035471 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:14.246073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:14.246036 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:15.787034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:15.787002 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:15.987925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:15.987895 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:16.195981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:16.195903 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:16.466204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:16.466125 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:17.947585 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:17.947557 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:19.581806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:19.581773 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:21.319765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:21.319731 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:22.731214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:22.731182 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:33:23.113720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:23.113640 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:23.335672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:23.335636 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:23.616734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:23.616684 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:23.941734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:23.941701 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:24.659601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:24.659570 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:24.874207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:24.874174 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:25.143906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:25.143870 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:27.086641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.086604 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:27.209783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.209748 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg"] Oct 30 13:33:27.210636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.210618 5515 cpu_manager.go:414] "containerMap: removing container" podUID="932ca447-138e-4e3d-80dd-786e0a6cf70d" containerName="payload-processing" Oct 30 13:33:27.210694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.210637 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932ca447-138e-4e3d-80dd-786e0a6cf70d" containerName="payload-processing" Oct 30 13:33:27.210807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.210797 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="932ca447-138e-4e3d-80dd-786e0a6cf70d" containerName="payload-processing" Oct 30 13:33:27.215951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.215932 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.254684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.254646 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg"] Oct 30 13:33:27.258904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.258872 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-e2e--d83bf907-dockercfg-2glgh\"" Oct 30 13:33:27.258904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.258893 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"agc-cred-c4446a02dec9\"" Oct 30 13:33:27.259123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.258876 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"kube-root-ca.crt\"" Oct 30 13:33:27.259123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.258924 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"openshift-service-ca.crt\"" Oct 30 13:33:27.259123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.258944 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0505b0\"" Oct 30 13:33:27.259123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.258961 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"routing-overlay\"" Oct 30 13:33:27.296562 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.296529 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:27.396375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.379355 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.398368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.398336 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.398660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.398642 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.398794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.398779 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxnql\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-kube-api-access-kxnql\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.500244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.500209 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.500458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.500252 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.500458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.500348 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.500458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.500382 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxnql\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-kube-api-access-kxnql\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.501099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.501077 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.501198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.501177 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.502677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.502656 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.524153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.524127 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxnql\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-kube-api-access-kxnql\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.534410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.534389 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:27.844465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:27.844434 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg"] Oct 30 13:33:27.845040 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:33:27.845016 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752452fe_2a48_4880_b564_38b09d8864e2.slice/crio-c40c4e3e7bcbcf3005f05290767ae1b1bd9d9e7cee1e9affb746619762c68ac9 WatchSource:0}: Error finding container c40c4e3e7bcbcf3005f05290767ae1b1bd9d9e7cee1e9affb746619762c68ac9: Status 404 returned error can't find the container with id c40c4e3e7bcbcf3005f05290767ae1b1bd9d9e7cee1e9affb746619762c68ac9 Oct 30 13:33:28.028294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:28.028091 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:28.451599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:28.451559 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" event={"ID":"752452fe-2a48-4880-b564-38b09d8864e2","Type":"ContainerStarted","Data":"2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf"} Oct 30 13:33:28.451992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:28.451613 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" event={"ID":"752452fe-2a48-4880-b564-38b09d8864e2","Type":"ContainerStarted","Data":"c40c4e3e7bcbcf3005f05290767ae1b1bd9d9e7cee1e9affb746619762c68ac9"} Oct 30 13:33:28.451992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:28.451779 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:28.500156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:28.500106 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" podStartSLOduration=1.5000746870000001 podStartE2EDuration="1.500074687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:28.496626107 +0000 UTC m=+3626.623427006" watchObservedRunningTime="2026-10-30 13:33:28.500074687 +0000 UTC m=+3626.626875602" Oct 30 13:33:28.934677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:28.934646 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:28.990198 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:33:28.990164 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:33:29.239680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:29.239606 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:29.463718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:29.463688 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:30.467123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.467094 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp"] Oct 30 13:33:30.472154 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.472134 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.487946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.487925 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"kube-root-ca.crt\"" Oct 30 13:33:30.492743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.492715 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp"] Oct 30 13:33:30.499178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.499154 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"agc-cred-c30a94579b6b\"" Oct 30 13:33:30.499314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.499214 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"payload-processing-external-model-e2e--69d1522b-dockercfg-fjjbd\"" Oct 30 13:33:30.499314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.499262 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-88e9f3\"" Oct 30 13:33:30.499448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.499326 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"routing-overlay\"" Oct 30 13:33:30.499448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.499165 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"openshift-service-ca.crt\"" Oct 30 13:33:30.499448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.499351 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-88e9f3\"/\"agc-cred-e4ee00a4a79c\"" Oct 30 13:33:30.562133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.562101 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.562310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.562229 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wmk\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-kube-api-access-j2wmk\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.562366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.562349 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.562488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.562472 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.663741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.663708 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.663939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.663848 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.664025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.663958 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wmk\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-kube-api-access-j2wmk\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.664127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.664030 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.664717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.664685 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.664717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.664686 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.666439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.666411 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.688074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.688042 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wmk\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-kube-api-access-j2wmk\") pod \"payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.793008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.792230 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:30.804397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:30.804353 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:31.114997 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:33:31.114954 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43607300_6cba_4eba_8b8c_7e54b9d60965.slice/crio-afd5b61e2f1481d119a6dbab0d09f1cd881f9ef460faee32517efcd0278bb29b WatchSource:0}: Error finding container afd5b61e2f1481d119a6dbab0d09f1cd881f9ef460faee32517efcd0278bb29b: Status 404 returned error can't find the container with id afd5b61e2f1481d119a6dbab0d09f1cd881f9ef460faee32517efcd0278bb29b Oct 30 13:33:31.115137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:31.114995 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp"] Oct 30 13:33:31.477529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:31.477487 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" event={"ID":"43607300-6cba-4eba-8b8c-7e54b9d60965","Type":"ContainerStarted","Data":"60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8"} Oct 30 13:33:31.477989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:31.477578 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" event={"ID":"43607300-6cba-4eba-8b8c-7e54b9d60965","Type":"ContainerStarted","Data":"afd5b61e2f1481d119a6dbab0d09f1cd881f9ef460faee32517efcd0278bb29b"} Oct 30 13:33:31.477989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:31.477691 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:31.529119 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:31.529069 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" podStartSLOduration=1.529055031 podStartE2EDuration="1.529055031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:31.523632968 +0000 UTC m=+3629.650433866" watchObservedRunningTime="2026-10-30 13:33:31.529055031 +0000 UTC m=+3629.655855929" Oct 30 13:33:32.770555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:32.770523 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:32.808719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:32.808687 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg"] Oct 30 13:33:32.811918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:32.811894 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:33.032037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:33.031964 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:33.274289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:33.272724 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:33.499512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:33.499475 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:34.030648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:34.030609 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:34.729630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:34.729596 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:36.407672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:36.407641 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:36.645294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:36.644628 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:36.856178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:36.856077 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:37.808962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:37.808901 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" podUID="752452fe-2a48-4880-b564-38b09d8864e2" containerName="payload-processing" containerID="cri-o://2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf" gracePeriod=25 Oct 30 13:33:38.110189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.110165 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:38.203592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.203556 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-kube-api-access-kxnql\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-kube-api-access-kxnql\") pod \"752452fe-2a48-4880-b564-38b09d8864e2\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " Oct 30 13:33:38.203592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.203593 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-plugins-config-volume\") pod \"752452fe-2a48-4880-b564-38b09d8864e2\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " Oct 30 13:33:38.203858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.203808 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-provider-credentials\") pod \"752452fe-2a48-4880-b564-38b09d8864e2\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " Oct 30 13:33:38.204074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.204046 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-routing-overlay-volume\") pod \"752452fe-2a48-4880-b564-38b09d8864e2\" (UID: \"752452fe-2a48-4880-b564-38b09d8864e2\") " Oct 30 13:33:38.204539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.204511 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-routing-overlay-volume" pod "752452fe-2a48-4880-b564-38b09d8864e2" (UID: "752452fe-2a48-4880-b564-38b09d8864e2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:38.204772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.204753 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:38.204823 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.204800 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-plugins-config-volume" pod "752452fe-2a48-4880-b564-38b09d8864e2" (UID: "752452fe-2a48-4880-b564-38b09d8864e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:38.205999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.205967 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-kube-api-access-kxnql" pod "752452fe-2a48-4880-b564-38b09d8864e2" (UID: "752452fe-2a48-4880-b564-38b09d8864e2"). InnerVolumeSpecName "kube-api-access-kxnql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:38.206263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.206240 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-provider-credentials" pod "752452fe-2a48-4880-b564-38b09d8864e2" (UID: "752452fe-2a48-4880-b564-38b09d8864e2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:38.305904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.305874 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:38.305904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.305901 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxnql\" (UniqueName: \"kubernetes.io/projected/752452fe-2a48-4880-b564-38b09d8864e2-kube-api-access-kxnql\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:38.305904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.305911 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752452fe-2a48-4880-b564-38b09d8864e2-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:38.450784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.450740 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:38.534968 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.534937 5515 generic.go:356] "Generic (PLEG): container finished" podID="752452fe-2a48-4880-b564-38b09d8864e2" containerID="2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf" exitCode=0 Oct 30 13:33:38.535141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.534982 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" Oct 30 13:33:38.535141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.535019 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" event={"ID":"752452fe-2a48-4880-b564-38b09d8864e2","Type":"ContainerDied","Data":"2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf"} Oct 30 13:33:38.535141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.535076 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg" event={"ID":"752452fe-2a48-4880-b564-38b09d8864e2","Type":"ContainerDied","Data":"c40c4e3e7bcbcf3005f05290767ae1b1bd9d9e7cee1e9affb746619762c68ac9"} Oct 30 13:33:38.535141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.535104 5515 scope.go:122] "RemoveContainer" containerID="2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf" Oct 30 13:33:38.546206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.546185 5515 scope.go:122] "RemoveContainer" containerID="2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf" Oct 30 13:33:38.546493 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:33:38.546474 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf\": container with ID starting with 2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf not found: ID does not exist" containerID="2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf" Oct 30 13:33:38.546553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.546499 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf"} err="failed to get container status \"2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf\": rpc error: code = NotFound desc = could not find container \"2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf\": container with ID starting with 2b336577d26953e3e0617baba98cba478ce92593530f62fca1731c031cddefcf not found: ID does not exist" Oct 30 13:33:38.590007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.589981 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg"] Oct 30 13:33:38.593636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:38.593606 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-77cdft6pg"] Oct 30 13:33:39.958093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:39.958061 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:40.028938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:40.027084 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752452fe-2a48-4880-b564-38b09d8864e2" path="/var/lib/kubelet/pods/752452fe-2a48-4880-b564-38b09d8864e2/volumes" Oct 30 13:33:40.155454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:40.155425 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp"] Oct 30 13:33:40.158673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:40.158624 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:40.181195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:40.181167 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:40.503213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:40.503185 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:41.079046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:41.079008 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:41.298614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:41.298582 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:41.944594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:41.944553 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:43.817741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:43.817707 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:44.820589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:44.820556 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:45.107343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.107245 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:45.156065 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.156021 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" podUID="43607300-6cba-4eba-8b8c-7e54b9d60965" containerName="payload-processing" containerID="cri-o://60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8" gracePeriod=25 Oct 30 13:33:45.437327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.437290 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:45.535488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.535457 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-provider-credentials\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-provider-credentials\") pod \"43607300-6cba-4eba-8b8c-7e54b9d60965\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " Oct 30 13:33:45.535680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.535519 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-kube-api-access-j2wmk\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-kube-api-access-j2wmk\") pod \"43607300-6cba-4eba-8b8c-7e54b9d60965\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " Oct 30 13:33:45.535680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.535592 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-routing-overlay-volume\") pod \"43607300-6cba-4eba-8b8c-7e54b9d60965\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " Oct 30 13:33:45.535680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.535634 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-plugins-config-volume\") pod \"43607300-6cba-4eba-8b8c-7e54b9d60965\" (UID: \"43607300-6cba-4eba-8b8c-7e54b9d60965\") " Oct 30 13:33:45.536040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.535984 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-routing-overlay-volume" pod "43607300-6cba-4eba-8b8c-7e54b9d60965" (UID: "43607300-6cba-4eba-8b8c-7e54b9d60965"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:45.536215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.536112 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-plugins-config-volume" pod "43607300-6cba-4eba-8b8c-7e54b9d60965" (UID: "43607300-6cba-4eba-8b8c-7e54b9d60965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:33:45.536354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.536284 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:45.536354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.536308 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43607300-6cba-4eba-8b8c-7e54b9d60965-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:45.537794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.537774 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-provider-credentials" pod "43607300-6cba-4eba-8b8c-7e54b9d60965" (UID: "43607300-6cba-4eba-8b8c-7e54b9d60965"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:45.538077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.538056 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-kube-api-access-j2wmk" pod "43607300-6cba-4eba-8b8c-7e54b9d60965" (UID: "43607300-6cba-4eba-8b8c-7e54b9d60965"). InnerVolumeSpecName "kube-api-access-j2wmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:45.591383 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.591324 5515 generic.go:356] "Generic (PLEG): container finished" podID="43607300-6cba-4eba-8b8c-7e54b9d60965" containerID="60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8" exitCode=0 Oct 30 13:33:45.591573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.591444 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" Oct 30 13:33:45.591640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.591569 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" event={"ID":"43607300-6cba-4eba-8b8c-7e54b9d60965","Type":"ContainerDied","Data":"60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8"} Oct 30 13:33:45.591697 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.591656 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp" event={"ID":"43607300-6cba-4eba-8b8c-7e54b9d60965","Type":"ContainerDied","Data":"afd5b61e2f1481d119a6dbab0d09f1cd881f9ef460faee32517efcd0278bb29b"} Oct 30 13:33:45.591933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.591892 5515 scope.go:122] "RemoveContainer" containerID="60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8" Oct 30 13:33:45.605526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.605508 5515 scope.go:122] "RemoveContainer" containerID="60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8" Oct 30 13:33:45.605819 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:33:45.605797 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8\": container with ID starting with 60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8 not found: ID does not exist" containerID="60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8" Oct 30 13:33:45.605889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.605830 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8"} err="failed to get container status \"60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8\": rpc error: code = NotFound desc = could not find container \"60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8\": container with ID starting with 60f2bd44bf67a47099028cc110db1dc926df31df5d81dd63057c153b559832d8 not found: ID does not exist" Oct 30 13:33:45.637905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.637876 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2wmk\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-kube-api-access-j2wmk\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:45.637905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.637905 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43607300-6cba-4eba-8b8c-7e54b9d60965-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:33:45.648676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.648650 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp"] Oct 30 13:33:45.654186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.654120 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-88e9f3/payload-processing-external-model-e2e-extmod-b-88e9f3-754cjgsgp"] Oct 30 13:33:45.765443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:45.765405 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:46.005094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.005017 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43607300-6cba-4eba-8b8c-7e54b9d60965" path="/var/lib/kubelet/pods/43607300-6cba-4eba-8b8c-7e54b9d60965/volumes" Oct 30 13:33:46.500962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.500929 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:46.503514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.503488 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx"] Oct 30 13:33:46.504574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.504549 5515 cpu_manager.go:414] "containerMap: removing container" podUID="43607300-6cba-4eba-8b8c-7e54b9d60965" containerName="payload-processing" Oct 30 13:33:46.504574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.504572 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43607300-6cba-4eba-8b8c-7e54b9d60965" containerName="payload-processing" Oct 30 13:33:46.504733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.504609 5515 cpu_manager.go:414] "containerMap: removing container" podUID="752452fe-2a48-4880-b564-38b09d8864e2" containerName="payload-processing" Oct 30 13:33:46.504733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.504615 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752452fe-2a48-4880-b564-38b09d8864e2" containerName="payload-processing" Oct 30 13:33:46.504807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.504745 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="752452fe-2a48-4880-b564-38b09d8864e2" containerName="payload-processing" Oct 30 13:33:46.504807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.504760 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43607300-6cba-4eba-8b8c-7e54b9d60965" containerName="payload-processing" Oct 30 13:33:46.510116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.510098 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:46.527835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.527809 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx"] Oct 30 13:33:46.535681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.535659 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9cc0e6\"" Oct 30 13:33:46.535817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.535789 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9cc0e6-dockercfg-zhxv6\"" Oct 30 13:33:46.551386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.551309 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2w2\" (UniqueName: \"kubernetes.io/projected/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-kube-api-access-hc2w2\") pod \"payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx\" (UID: \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:46.551546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.551526 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx\" (UID: \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:46.570392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.570367 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf"] Oct 30 13:33:46.575049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.575030 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:46.619467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.611242 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf"] Oct 30 13:33:46.653291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.652955 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf045354-c623-4b61-bbd9-ab57faeb18a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf\" (UID: \"bf045354-c623-4b61-bbd9-ab57faeb18a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:46.653291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.653171 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2w2\" (UniqueName: \"kubernetes.io/projected/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-kube-api-access-hc2w2\") pod \"payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx\" (UID: \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:46.653577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.653304 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w27fw\" (UniqueName: \"kubernetes.io/projected/bf045354-c623-4b61-bbd9-ab57faeb18a2-kube-api-access-w27fw\") pod \"payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf\" (UID: \"bf045354-c623-4b61-bbd9-ab57faeb18a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:46.653577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.653387 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx\" (UID: \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:46.678292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.654603 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx\" (UID: \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:46.719322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.708623 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2w2\" (UniqueName: \"kubernetes.io/projected/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-kube-api-access-hc2w2\") pod \"payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx\" (UID: \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:46.755336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.754924 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w27fw\" (UniqueName: \"kubernetes.io/projected/bf045354-c623-4b61-bbd9-ab57faeb18a2-kube-api-access-w27fw\") pod \"payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf\" (UID: \"bf045354-c623-4b61-bbd9-ab57faeb18a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:46.755336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.755092 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf045354-c623-4b61-bbd9-ab57faeb18a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf\" (UID: \"bf045354-c623-4b61-bbd9-ab57faeb18a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:46.756320 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.756231 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf045354-c623-4b61-bbd9-ab57faeb18a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf\" (UID: \"bf045354-c623-4b61-bbd9-ab57faeb18a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:46.788560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.786575 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w27fw\" (UniqueName: \"kubernetes.io/projected/bf045354-c623-4b61-bbd9-ab57faeb18a2-kube-api-access-w27fw\") pod \"payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf\" (UID: \"bf045354-c623-4b61-bbd9-ab57faeb18a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:46.817616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.817586 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:46.825557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.825530 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:46.890339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:46.890246 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:47.149292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.149239 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx"] Oct 30 13:33:47.150609 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:33:47.150579 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff4dd5f_793d_4148_8be2_8cbe26c2eddf.slice/crio-29da12fb3b764f4da23e9532e899faee8bba8cec266381c1179a62721b691a12 WatchSource:0}: Error finding container 29da12fb3b764f4da23e9532e899faee8bba8cec266381c1179a62721b691a12: Status 404 returned error can't find the container with id 29da12fb3b764f4da23e9532e899faee8bba8cec266381c1179a62721b691a12 Oct 30 13:33:47.237153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.237126 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf"] Oct 30 13:33:47.238070 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:33:47.238046 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf045354_c623_4b61_bbd9_ab57faeb18a2.slice/crio-e86207952643ad1ad8707af2acdd05936d139578726b1a63846a205638d9697a WatchSource:0}: Error finding container e86207952643ad1ad8707af2acdd05936d139578726b1a63846a205638d9697a: Status 404 returned error can't find the container with id e86207952643ad1ad8707af2acdd05936d139578726b1a63846a205638d9697a Oct 30 13:33:47.634984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.634911 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" event={"ID":"aff4dd5f-793d-4148-8be2-8cbe26c2eddf","Type":"ContainerStarted","Data":"d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231"} Oct 30 13:33:47.634984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.634969 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" event={"ID":"aff4dd5f-793d-4148-8be2-8cbe26c2eddf","Type":"ContainerStarted","Data":"29da12fb3b764f4da23e9532e899faee8bba8cec266381c1179a62721b691a12"} Oct 30 13:33:47.635295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.635068 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:47.637063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.637034 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" event={"ID":"bf045354-c623-4b61-bbd9-ab57faeb18a2","Type":"ContainerStarted","Data":"b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13"} Oct 30 13:33:47.637199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.637098 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" event={"ID":"bf045354-c623-4b61-bbd9-ab57faeb18a2","Type":"ContainerStarted","Data":"e86207952643ad1ad8707af2acdd05936d139578726b1a63846a205638d9697a"} Oct 30 13:33:47.637199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.637130 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:47.686824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.686775 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" podStartSLOduration=1.6867610659999999 podStartE2EDuration="1.686761066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:47.682493092 +0000 UTC m=+3645.809294189" watchObservedRunningTime="2026-10-30 13:33:47.686761066 +0000 UTC m=+3645.813561999" Oct 30 13:33:47.817318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:47.817240 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:49.411693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:49.411657 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:51.141491 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:51.141458 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:52.905179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:52.905090 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:54.739557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:54.739512 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:56.579579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:56.579545 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:58.305527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:58.305491 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:33:58.647908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:58.647880 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:33:58.648099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:58.647950 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:33:58.696703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:33:58.696654 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" podStartSLOduration=12.696639802 podStartE2EDuration="12.696639802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:47.731174077 +0000 UTC m=+3645.857974973" watchObservedRunningTime="2026-10-30 13:33:58.696639802 +0000 UTC m=+3656.823440733" Oct 30 13:33:58.989693 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:33:58.989614 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:34:00.228369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:00.228315 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:00.336200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:00.336156 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5"] Oct 30 13:34:00.503870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:00.503779 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:00.747876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:00.747837 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:00.977613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:00.977577 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:01.407617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:01.407579 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:01.989315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:01.989241 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:03.643983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:03.643945 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:05.337197 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.337149 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" podUID="689fb278-7e18-491e-94c2-2e37cf52a896" containerName="payload-processing" containerID="cri-o://4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a" gracePeriod=25 Oct 30 13:34:05.509685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.509652 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:05.629175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.629143 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:34:05.698028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.697991 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-routing-overlay-volume\") pod \"689fb278-7e18-491e-94c2-2e37cf52a896\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " Oct 30 13:34:05.698248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.698199 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-kube-api-access-7dvg7\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-kube-api-access-7dvg7\") pod \"689fb278-7e18-491e-94c2-2e37cf52a896\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " Oct 30 13:34:05.698389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.698293 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-provider-credentials\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-provider-credentials\") pod \"689fb278-7e18-491e-94c2-2e37cf52a896\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " Oct 30 13:34:05.698455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.698387 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-plugins-config-volume\") pod \"689fb278-7e18-491e-94c2-2e37cf52a896\" (UID: \"689fb278-7e18-491e-94c2-2e37cf52a896\") " Oct 30 13:34:05.699692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.699662 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-plugins-config-volume" pod "689fb278-7e18-491e-94c2-2e37cf52a896" (UID: "689fb278-7e18-491e-94c2-2e37cf52a896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:05.699861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.699710 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-routing-overlay-volume" pod "689fb278-7e18-491e-94c2-2e37cf52a896" (UID: "689fb278-7e18-491e-94c2-2e37cf52a896"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:05.702024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.702002 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-kube-api-access-7dvg7" pod "689fb278-7e18-491e-94c2-2e37cf52a896" (UID: "689fb278-7e18-491e-94c2-2e37cf52a896"). InnerVolumeSpecName "kube-api-access-7dvg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:05.702166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.702150 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-provider-credentials" pod "689fb278-7e18-491e-94c2-2e37cf52a896" (UID: "689fb278-7e18-491e-94c2-2e37cf52a896"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:05.801193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.801164 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dvg7\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-kube-api-access-7dvg7\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:34:05.801193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.801191 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/689fb278-7e18-491e-94c2-2e37cf52a896-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:34:05.801193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.801202 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:34:05.801493 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.801210 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/689fb278-7e18-491e-94c2-2e37cf52a896-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:34:05.806191 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.806161 5515 generic.go:356] "Generic (PLEG): container finished" podID="689fb278-7e18-491e-94c2-2e37cf52a896" containerID="4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a" exitCode=0 Oct 30 13:34:05.806364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.806218 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" Oct 30 13:34:05.806364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.806242 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" event={"ID":"689fb278-7e18-491e-94c2-2e37cf52a896","Type":"ContainerDied","Data":"4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a"} Oct 30 13:34:05.806364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.806343 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5" event={"ID":"689fb278-7e18-491e-94c2-2e37cf52a896","Type":"ContainerDied","Data":"7ef6818fb4ec64949ead0def7a0f9fbd5d1d6b782829387c445f27e4c351819a"} Oct 30 13:34:05.806520 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.806373 5515 scope.go:122] "RemoveContainer" containerID="4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a" Oct 30 13:34:05.817635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.817595 5515 scope.go:122] "RemoveContainer" containerID="4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a" Oct 30 13:34:05.817875 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:34:05.817856 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a\": container with ID starting with 4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a not found: ID does not exist" containerID="4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a" Oct 30 13:34:05.817926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.817883 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a"} err="failed to get container status \"4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a\": rpc error: code = NotFound desc = could not find container \"4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a\": container with ID starting with 4ab1c5e4d3325006cd4cff6ab3db618879b0d3412907de0f6824f11a8fda2c2a not found: ID does not exist" Oct 30 13:34:05.864602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.864534 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5"] Oct 30 13:34:05.872243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:05.872218 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-dc7cjfpm5"] Oct 30 13:34:06.003580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:06.003545 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="689fb278-7e18-491e-94c2-2e37cf52a896" path="/var/lib/kubelet/pods/689fb278-7e18-491e-94c2-2e37cf52a896/volumes" Oct 30 13:34:07.273122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:07.273074 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:07.591079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:07.590988 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:07.810637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:07.810606 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:08.101939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:08.101909 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:08.336670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:08.336623 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:08.615214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:08.615172 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:08.838607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:08.838575 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:09.105679 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:09.105645 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:09.434635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:09.434601 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:09.714710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:09.714629 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:11.276223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:11.276186 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:13.042875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:13.042838 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:13.457533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:13.457503 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:13.694060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:13.694027 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:13.917701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:13.917666 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:14.124427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:14.124395 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:15.175370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.175334 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:15.487636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.487555 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:15.559913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.559880 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm"] Oct 30 13:34:15.560743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.560727 5515 cpu_manager.go:414] "containerMap: removing container" podUID="689fb278-7e18-491e-94c2-2e37cf52a896" containerName="payload-processing" Oct 30 13:34:15.560800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.560744 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689fb278-7e18-491e-94c2-2e37cf52a896" containerName="payload-processing" Oct 30 13:34:15.560901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.560892 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="689fb278-7e18-491e-94c2-2e37cf52a896" containerName="payload-processing" Oct 30 13:34:15.565943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.565922 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:15.585618 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.585595 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm"] Oct 30 13:34:15.595546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.595517 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ba646b\"" Oct 30 13:34:15.605995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.605965 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ba646b-dockercfg-75slr\"" Oct 30 13:34:15.631184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.630786 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f318bc-f058-4cd0-b681-75f77bffa1f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm\" (UID: \"67f318bc-f058-4cd0-b681-75f77bffa1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:15.631184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.631015 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tncd\" (UniqueName: \"kubernetes.io/projected/67f318bc-f058-4cd0-b681-75f77bffa1f0-kube-api-access-9tncd\") pod \"payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm\" (UID: \"67f318bc-f058-4cd0-b681-75f77bffa1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:15.728290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.696438 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc"] Oct 30 13:34:15.750693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.750613 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:15.753087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.753042 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tncd\" (UniqueName: \"kubernetes.io/projected/67f318bc-f058-4cd0-b681-75f77bffa1f0-kube-api-access-9tncd\") pod \"payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm\" (UID: \"67f318bc-f058-4cd0-b681-75f77bffa1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:15.753840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.753638 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f318bc-f058-4cd0-b681-75f77bffa1f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm\" (UID: \"67f318bc-f058-4cd0-b681-75f77bffa1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:15.755172 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.755130 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f318bc-f058-4cd0-b681-75f77bffa1f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm\" (UID: \"67f318bc-f058-4cd0-b681-75f77bffa1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:15.782285 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.781438 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:15.801367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.800976 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc"] Oct 30 13:34:15.809210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.804938 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tncd\" (UniqueName: \"kubernetes.io/projected/67f318bc-f058-4cd0-b681-75f77bffa1f0-kube-api-access-9tncd\") pod \"payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm\" (UID: \"67f318bc-f058-4cd0-b681-75f77bffa1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:15.855260 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.855210 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlr6j\" (UniqueName: \"kubernetes.io/projected/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-kube-api-access-vlr6j\") pod \"payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc\" (UID: \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:15.855468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.855354 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc\" (UID: \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:15.881608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.881577 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:15.956638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.956599 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlr6j\" (UniqueName: \"kubernetes.io/projected/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-kube-api-access-vlr6j\") pod \"payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc\" (UID: \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:15.956787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.956655 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc\" (UID: \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:15.961382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.961350 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc\" (UID: \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:15.982067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:15.981511 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlr6j\" (UniqueName: \"kubernetes.io/projected/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-kube-api-access-vlr6j\") pod \"payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc\" (UID: \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:16.137209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.137178 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:16.190197 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.190166 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm"] Oct 30 13:34:16.192669 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:34:16.192568 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f318bc_f058_4cd0_b681_75f77bffa1f0.slice/crio-669161a52ae26252a3482f27e9e8ab6ea33bcbcbc3b2833be4418d1a3d0a6775 WatchSource:0}: Error finding container 669161a52ae26252a3482f27e9e8ab6ea33bcbcbc3b2833be4418d1a3d0a6775: Status 404 returned error can't find the container with id 669161a52ae26252a3482f27e9e8ab6ea33bcbcbc3b2833be4418d1a3d0a6775 Oct 30 13:34:16.454673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.454644 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc"] Oct 30 13:34:16.455926 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:34:16.455898 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d92fb0_19e2_47d5_a770_069c5a1f2d4d.slice/crio-cfd159ebff583f1f7db0c3cc5c0fb48ac605289f38bedd124ca381e075a856a9 WatchSource:0}: Error finding container cfd159ebff583f1f7db0c3cc5c0fb48ac605289f38bedd124ca381e075a856a9: Status 404 returned error can't find the container with id cfd159ebff583f1f7db0c3cc5c0fb48ac605289f38bedd124ca381e075a856a9 Oct 30 13:34:16.895156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.895121 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" event={"ID":"67f318bc-f058-4cd0-b681-75f77bffa1f0","Type":"ContainerStarted","Data":"831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f"} Oct 30 13:34:16.895375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.895184 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" event={"ID":"67f318bc-f058-4cd0-b681-75f77bffa1f0","Type":"ContainerStarted","Data":"669161a52ae26252a3482f27e9e8ab6ea33bcbcbc3b2833be4418d1a3d0a6775"} Oct 30 13:34:16.895375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.895223 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:16.896923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.896895 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" event={"ID":"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d","Type":"ContainerStarted","Data":"342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285"} Oct 30 13:34:16.897032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.896944 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" event={"ID":"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d","Type":"ContainerStarted","Data":"cfd159ebff583f1f7db0c3cc5c0fb48ac605289f38bedd124ca381e075a856a9"} Oct 30 13:34:16.897032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.896975 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:16.938885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.938839 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" podStartSLOduration=1.9388275830000001 podStartE2EDuration="1.938827583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:16.93565338 +0000 UTC m=+3675.062454277" watchObservedRunningTime="2026-10-30 13:34:16.938827583 +0000 UTC m=+3675.065628479" Oct 30 13:34:16.989558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.989502 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" podStartSLOduration=1.989489545 podStartE2EDuration="1.989489545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:16.988018501 +0000 UTC m=+3675.114819399" watchObservedRunningTime="2026-10-30 13:34:16.989489545 +0000 UTC m=+3675.116290441" Oct 30 13:34:16.999094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:16.999046 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:17.275858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.275783 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:17.362750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.362720 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm"] Oct 30 13:34:17.367665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.367650 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:17.384546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.384520 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-22efcf\"" Oct 30 13:34:17.385520 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.385499 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-22efcf-dockercfg-bfscs\"" Oct 30 13:34:17.389443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.389421 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm"] Oct 30 13:34:17.485304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.484317 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh"] Oct 30 13:34:17.486703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.486663 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm\" (UID: \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:17.486866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.486795 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5hk\" (UniqueName: \"kubernetes.io/projected/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-kube-api-access-ms5hk\") pod \"payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm\" (UID: \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:17.490046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.490024 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:17.490330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.490050 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh"] Oct 30 13:34:17.491112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.491091 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:17.588369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.588248 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380c7464-dcec-4c9a-9bce-d54738f6073e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh\" (UID: \"380c7464-dcec-4c9a-9bce-d54738f6073e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:17.588597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.588569 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mg8\" (UniqueName: \"kubernetes.io/projected/380c7464-dcec-4c9a-9bce-d54738f6073e-kube-api-access-68mg8\") pod \"payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh\" (UID: \"380c7464-dcec-4c9a-9bce-d54738f6073e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:17.588680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.588631 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm\" (UID: \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:17.588810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.588786 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5hk\" (UniqueName: \"kubernetes.io/projected/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-kube-api-access-ms5hk\") pod \"payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm\" (UID: \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:17.589684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.589660 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm\" (UID: \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:17.612832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.612799 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5hk\" (UniqueName: \"kubernetes.io/projected/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-kube-api-access-ms5hk\") pod \"payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm\" (UID: \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:17.684383 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.684353 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:17.690241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.690214 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mg8\" (UniqueName: \"kubernetes.io/projected/380c7464-dcec-4c9a-9bce-d54738f6073e-kube-api-access-68mg8\") pod \"payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh\" (UID: \"380c7464-dcec-4c9a-9bce-d54738f6073e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:17.690414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.690396 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380c7464-dcec-4c9a-9bce-d54738f6073e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh\" (UID: \"380c7464-dcec-4c9a-9bce-d54738f6073e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:17.691282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.691240 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380c7464-dcec-4c9a-9bce-d54738f6073e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh\" (UID: \"380c7464-dcec-4c9a-9bce-d54738f6073e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:17.717141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.717099 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mg8\" (UniqueName: \"kubernetes.io/projected/380c7464-dcec-4c9a-9bce-d54738f6073e-kube-api-access-68mg8\") pod \"payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh\" (UID: \"380c7464-dcec-4c9a-9bce-d54738f6073e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:17.822634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:17.822590 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:18.003418 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:34:18.003390 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca90c36_dd12_47b0_a6d7_d01f0d79ab7c.slice/crio-010a24f7deb97875761f149c1af927661d427aa304d5c6aa959260ccb4f07232 WatchSource:0}: Error finding container 010a24f7deb97875761f149c1af927661d427aa304d5c6aa959260ccb4f07232: Status 404 returned error can't find the container with id 010a24f7deb97875761f149c1af927661d427aa304d5c6aa959260ccb4f07232 Oct 30 13:34:18.003966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.003948 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm"] Oct 30 13:34:18.151625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.151600 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh"] Oct 30 13:34:18.152460 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:34:18.152435 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380c7464_dcec_4c9a_9bce_d54738f6073e.slice/crio-5e39303eb0589a09bd5e273766d9a6bcdfda92a6a2deafa449b0cf26fe5ebf9b WatchSource:0}: Error finding container 5e39303eb0589a09bd5e273766d9a6bcdfda92a6a2deafa449b0cf26fe5ebf9b: Status 404 returned error can't find the container with id 5e39303eb0589a09bd5e273766d9a6bcdfda92a6a2deafa449b0cf26fe5ebf9b Oct 30 13:34:18.949743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.949707 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" event={"ID":"380c7464-dcec-4c9a-9bce-d54738f6073e","Type":"ContainerStarted","Data":"7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8"} Oct 30 13:34:18.950394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.949770 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" event={"ID":"380c7464-dcec-4c9a-9bce-d54738f6073e","Type":"ContainerStarted","Data":"5e39303eb0589a09bd5e273766d9a6bcdfda92a6a2deafa449b0cf26fe5ebf9b"} Oct 30 13:34:18.950394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.949820 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:18.951743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.951710 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" event={"ID":"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c","Type":"ContainerStarted","Data":"244cfd869ada591cb80cfb5a60affd32a4d67c95fd7b567f69c37c7ab06372cc"} Oct 30 13:34:18.951886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.951763 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" event={"ID":"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c","Type":"ContainerStarted","Data":"010a24f7deb97875761f149c1af927661d427aa304d5c6aa959260ccb4f07232"} Oct 30 13:34:18.951886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.951861 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:18.969525 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.969490 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:18.993262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:18.993198 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" podStartSLOduration=1.993185602 podStartE2EDuration="1.993185602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:18.990431986 +0000 UTC m=+3677.117232883" watchObservedRunningTime="2026-10-30 13:34:18.993185602 +0000 UTC m=+3677.119986494" Oct 30 13:34:19.035240 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:19.035191 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" podStartSLOduration=2.03517732 podStartE2EDuration="2.03517732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:19.031023531 +0000 UTC m=+3677.157824428" watchObservedRunningTime="2026-10-30 13:34:19.03517732 +0000 UTC m=+3677.161978217" Oct 30 13:34:20.618712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:20.618671 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:22.447702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:22.447667 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:24.255525 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:24.255493 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:25.834731 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:25.834689 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:27.441780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:27.441729 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:27.501566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:27.501532 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:34:27.921221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:27.921193 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:34:27.921432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:27.921310 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:34:28.988810 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:34:28.988777 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:34:29.299847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:29.299768 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:29.962611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:29.962586 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:34:29.962996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:29.962981 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:34:31.029313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:31.029244 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:32.466713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.466677 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv"] Oct 30 13:34:32.472646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.472630 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.499346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.499319 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9cc0e6-serving-cert\"" Oct 30 13:34:32.504428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.504399 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv"] Oct 30 13:34:32.544216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.544189 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8130ae3f-6e4d-428d-a10c-b830ee51e6da-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.544425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.544405 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-maas-api-tls\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.544482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.544450 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-metrics-certs\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.544645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.544626 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb9ck\" (UniqueName: \"kubernetes.io/projected/8130ae3f-6e4d-428d-a10c-b830ee51e6da-kube-api-access-qb9ck\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.646517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.646487 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb9ck\" (UniqueName: \"kubernetes.io/projected/8130ae3f-6e4d-428d-a10c-b830ee51e6da-kube-api-access-qb9ck\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.646677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.646551 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8130ae3f-6e4d-428d-a10c-b830ee51e6da-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.646718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.646693 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-maas-api-tls\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.646751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.646742 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-metrics-certs\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.647376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.647353 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8130ae3f-6e4d-428d-a10c-b830ee51e6da-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.649209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.649185 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-metrics-certs\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.649308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.649223 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-maas-api-tls\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.679292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.677906 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb9ck\" (UniqueName: \"kubernetes.io/projected/8130ae3f-6e4d-428d-a10c-b830ee51e6da-kube-api-access-qb9ck\") pod \"maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.690461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.683390 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:32.788161 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.788065 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:32.897598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:32.897566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:33.101309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:33.101236 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv"] Oct 30 13:34:33.102375 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:34:33.102342 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8130ae3f_6e4d_428d_a10c_b830ee51e6da.slice/crio-8e661abd8d122fa1ab05f8b389307487dedf4f1c50c509cbfc75a52bbe6a92eb WatchSource:0}: Error finding container 8e661abd8d122fa1ab05f8b389307487dedf4f1c50c509cbfc75a52bbe6a92eb: Status 404 returned error can't find the container with id 8e661abd8d122fa1ab05f8b389307487dedf4f1c50c509cbfc75a52bbe6a92eb Oct 30 13:34:33.104235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:33.104216 5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:34:34.071732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:34.071699 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" event={"ID":"8130ae3f-6e4d-428d-a10c-b830ee51e6da","Type":"ContainerStarted","Data":"213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142"} Oct 30 13:34:34.072132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:34.071752 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" event={"ID":"8130ae3f-6e4d-428d-a10c-b830ee51e6da","Type":"ContainerStarted","Data":"8e661abd8d122fa1ab05f8b389307487dedf4f1c50c509cbfc75a52bbe6a92eb"} Oct 30 13:34:34.072132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:34.071796 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:34.119501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:34.119452 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" podStartSLOduration=1.709103762 podStartE2EDuration="2.119437793s" totalImagesPullingTime="410.334031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:34.11554413 +0000 UTC m=+3692.242345032" watchObservedRunningTime="2026-10-30 13:34:34.119437793 +0000 UTC m=+3692.246238689" Oct 30 13:34:34.453490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:34.453447 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:36.198531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:36.198488 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:38.007847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:38.007813 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:39.723964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:39.723935 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:40.085442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:40.085362 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:34:41.472827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:41.472789 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:43.397716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:43.397633 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:45.128476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:45.128442 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:46.845382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:46.845347 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:48.608747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:48.608712 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:50.472889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:50.472857 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:52.151814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:52.151777 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:54.112875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:54.112840 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:55.710042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:55.710004 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:55.923108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:55.923076 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:56.153068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:56.153036 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:57.928099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:57.928069 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:34:58.989874 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:34:58.989832 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:34:59.775634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:34:59.775596 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:01.553920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:01.553889 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:03.144600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.144559 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv"] Oct 30 13:35:03.151349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.151326 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.177052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.177026 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ba646b-serving-cert\"" Oct 30 13:35:03.183210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.183181 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv"] Oct 30 13:35:03.312146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.312113 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-metrics-certs\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.312362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.312156 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-maas-api-tls\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.312362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.312302 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.312559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.312538 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfjk\" (UniqueName: \"kubernetes.io/projected/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-kube-api-access-rhfjk\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.346556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.346517 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:03.413980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.413895 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.414173 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.414090 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfjk\" (UniqueName: \"kubernetes.io/projected/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-kube-api-access-rhfjk\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.414280 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.414247 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-metrics-certs\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.414359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.414335 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-maas-api-tls\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.414738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.414714 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.417229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.417204 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-metrics-certs\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.419804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.419778 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-maas-api-tls\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.440239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.440208 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfjk\" (UniqueName: \"kubernetes.io/projected/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-kube-api-access-rhfjk\") pod \"maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.466918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.466896 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:03.569138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.568947 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:03.778552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.778524 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv"] Oct 30 13:35:03.779129 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:35:03.779103 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba10fcf2_1c37_46c7_ba6c_b3c0817407d6.slice/crio-3dd223d26397d52fea2bf751b942834d47491c4d60a71e0390ae327998e5852f WatchSource:0}: Error finding container 3dd223d26397d52fea2bf751b942834d47491c4d60a71e0390ae327998e5852f: Status 404 returned error can't find the container with id 3dd223d26397d52fea2bf751b942834d47491c4d60a71e0390ae327998e5852f Oct 30 13:35:03.797925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:03.797898 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:04.375297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:04.375207 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" event={"ID":"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6","Type":"ContainerStarted","Data":"0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5"} Oct 30 13:35:04.375297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:04.375258 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" event={"ID":"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6","Type":"ContainerStarted","Data":"3dd223d26397d52fea2bf751b942834d47491c4d60a71e0390ae327998e5852f"} Oct 30 13:35:04.375744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:04.375345 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:04.436218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:04.436163 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" podStartSLOduration=1.097674147 podStartE2EDuration="1.436145556s" totalImagesPullingTime="338.471409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:04.423809765 +0000 UTC m=+3722.550610666" watchObservedRunningTime="2026-10-30 13:35:04.436145556 +0000 UTC m=+3722.562946480" Oct 30 13:35:05.373869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:05.373834 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:05.590700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:05.590661 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:07.396628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:07.396595 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:09.118202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:09.118169 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:10.388873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:10.388844 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:35:10.872754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:10.872661 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:12.696420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:12.696386 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:14.308861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:14.308821 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:14.519489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:14.519457 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:16.114977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:16.114930 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:17.994061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:17.994009 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:19.227439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:19.227404 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:19.468796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:19.468759 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:19.894667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:19.894637 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:20.118310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:20.118284 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:20.344826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:20.344741 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:21.740726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:21.740695 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:23.288698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:23.288666 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:23.535639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:23.535608 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:23.778773 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:23.778738 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:24.155904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:24.155858 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:25.745236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:25.745190 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:25.960446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:25.960416 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:25.987102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:25.987075 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v"] Oct 30 13:35:26.181786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:26.181751 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:26.449664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:26.449548 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:26.948749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:26.948718 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs"] Oct 30 13:35:27.138992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:27.138960 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:27.371866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:27.371782 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:27.695699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:27.695662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:27.922063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:27.922025 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:28.249705 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:28.249666 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:28.486650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:28.486619 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:28.992161 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:35:28.992081 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:35:29.432107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:29.432046 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:30.987876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:30.987833 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" podUID="c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" containerName="payload-processing" containerID="cri-o://09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505" gracePeriod=25 Oct 30 13:35:31.268613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.268589 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:35:31.314377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.314351 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-kube-api-access-d7bhw\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-kube-api-access-d7bhw\") pod \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " Oct 30 13:35:31.314532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.314413 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-plugins-config-volume\") pod \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " Oct 30 13:35:31.314532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.314477 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-provider-credentials\") pod \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " Oct 30 13:35:31.314681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.314604 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-routing-overlay-volume\") pod \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\" (UID: \"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c\") " Oct 30 13:35:31.314962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.314915 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-plugins-config-volume" pod "c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" (UID: "c42d2f44-6c24-4dc6-bf69-6f2049f34a0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:31.314962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.314937 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-routing-overlay-volume" pod "c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" (UID: "c42d2f44-6c24-4dc6-bf69-6f2049f34a0c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:31.315169 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.315150 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.315169 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.315170 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.316602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.316584 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-kube-api-access-d7bhw" pod "c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" (UID: "c42d2f44-6c24-4dc6-bf69-6f2049f34a0c"). InnerVolumeSpecName "kube-api-access-d7bhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:31.316741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.316721 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-provider-credentials" pod "c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" (UID: "c42d2f44-6c24-4dc6-bf69-6f2049f34a0c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:31.416598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.416562 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7bhw\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-kube-api-access-d7bhw\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.416598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.416591 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:35:31.499952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.499821 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:31.633094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.633056 5515 generic.go:356] "Generic (PLEG): container finished" podID="c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" containerID="09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505" exitCode=0 Oct 30 13:35:31.633325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.633106 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" Oct 30 13:35:31.633325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.633135 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" event={"ID":"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c","Type":"ContainerDied","Data":"09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505"} Oct 30 13:35:31.633325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.633186 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v" event={"ID":"c42d2f44-6c24-4dc6-bf69-6f2049f34a0c","Type":"ContainerDied","Data":"6d0444eeda65839ae359383151f481f8f14d288e15bf3f61e5d9e4bcbf0cfce4"} Oct 30 13:35:31.633325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.633214 5515 scope.go:122] "RemoveContainer" containerID="09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505" Oct 30 13:35:31.644562 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.644543 5515 scope.go:122] "RemoveContainer" containerID="09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505" Oct 30 13:35:31.644836 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:35:31.644817 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505\": container with ID starting with 09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505 not found: ID does not exist" containerID="09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505" Oct 30 13:35:31.644889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.644844 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505"} err="failed to get container status \"09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505\": rpc error: code = NotFound desc = could not find container \"09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505\": container with ID starting with 09bd842d7e16901cad1344684619a1207af98fed4ad307efa639e16986851505 not found: ID does not exist" Oct 30 13:35:31.692701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.692669 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v"] Oct 30 13:35:31.696788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.696766 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5b69cwh4v"] Oct 30 13:35:31.949927 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:31.949889 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" podUID="755120a9-0431-406c-8f80-1f24374564cb" containerName="payload-processing" containerID="cri-o://e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c" gracePeriod=25 Oct 30 13:35:32.005659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.005626 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" path="/var/lib/kubelet/pods/c42d2f44-6c24-4dc6-bf69-6f2049f34a0c/volumes" Oct 30 13:35:32.247064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.247038 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:35:32.331156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.331125 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-routing-overlay-volume\") pod \"755120a9-0431-406c-8f80-1f24374564cb\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " Oct 30 13:35:32.331373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.331171 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-provider-credentials\") pod \"755120a9-0431-406c-8f80-1f24374564cb\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " Oct 30 13:35:32.331373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.331240 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-plugins-config-volume\") pod \"755120a9-0431-406c-8f80-1f24374564cb\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " Oct 30 13:35:32.331373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.331293 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-kube-api-access-qz9z7\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-kube-api-access-qz9z7\") pod \"755120a9-0431-406c-8f80-1f24374564cb\" (UID: \"755120a9-0431-406c-8f80-1f24374564cb\") " Oct 30 13:35:32.331770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.331612 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-routing-overlay-volume" pod "755120a9-0431-406c-8f80-1f24374564cb" (UID: "755120a9-0431-406c-8f80-1f24374564cb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:32.331879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.331827 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-plugins-config-volume" pod "755120a9-0431-406c-8f80-1f24374564cb" (UID: "755120a9-0431-406c-8f80-1f24374564cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:32.332798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.332768 5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-routing-overlay-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:35:32.332977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.332803 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755120a9-0431-406c-8f80-1f24374564cb-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:35:32.349309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.348163 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-kube-api-access-qz9z7" pod "755120a9-0431-406c-8f80-1f24374564cb" (UID: "755120a9-0431-406c-8f80-1f24374564cb"). InnerVolumeSpecName "kube-api-access-qz9z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:32.349309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.348216 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-provider-credentials" pod "755120a9-0431-406c-8f80-1f24374564cb" (UID: "755120a9-0431-406c-8f80-1f24374564cb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:32.363406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.363376 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:32.451640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.446729 5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-provider-credentials\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:35:32.451640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.446768 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz9z7\" (UniqueName: \"kubernetes.io/projected/755120a9-0431-406c-8f80-1f24374564cb-kube-api-access-qz9z7\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:35:32.641338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.641309 5515 generic.go:356] "Generic (PLEG): container finished" podID="755120a9-0431-406c-8f80-1f24374564cb" containerID="e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c" exitCode=0 Oct 30 13:35:32.641492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.641369 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" Oct 30 13:35:32.641492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.641394 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" event={"ID":"755120a9-0431-406c-8f80-1f24374564cb","Type":"ContainerDied","Data":"e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c"} Oct 30 13:35:32.641492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.641467 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs" event={"ID":"755120a9-0431-406c-8f80-1f24374564cb","Type":"ContainerDied","Data":"9ed8398abd03a9e274a410ca0b3683509c3e8c1fbec83734e187fd645787467f"} Oct 30 13:35:32.641669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.641496 5515 scope.go:122] "RemoveContainer" containerID="e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c" Oct 30 13:35:32.656875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.656851 5515 scope.go:122] "RemoveContainer" containerID="e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c" Oct 30 13:35:32.657255 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:35:32.657232 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c\": container with ID starting with e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c not found: ID does not exist" containerID="e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c" Oct 30 13:35:32.657383 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.657263 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c"} err="failed to get container status \"e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c\": rpc error: code = NotFound desc = could not find container \"e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c\": container with ID starting with e4d77bf922e8712a00974bc831f5a76951db6bdff569c29ae48b3028caebe00c not found: ID does not exist" Oct 30 13:35:32.658487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.658467 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:32.697690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.697661 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs"] Oct 30 13:35:32.703205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.703182 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-776575vcs"] Oct 30 13:35:32.895331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:32.895226 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:33.411193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:33.411167 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:33.638920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:33.638883 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:34.003825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:34.003794 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755120a9-0431-406c-8f80-1f24374564cb" path="/var/lib/kubelet/pods/755120a9-0431-406c-8f80-1f24374564cb/volumes" Oct 30 13:35:34.103541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:34.103503 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:34.353372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:34.348399 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:34.744244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:34.744202 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:34.975750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:34.975716 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:35.273496 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:35.273450 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:35.692553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:35.692522 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:35.972847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:35.972760 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:37.703335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:37.703262 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:37.942317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:37.942259 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:39.457214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.457179 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:39.579050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.579019 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx"] Oct 30 13:35:39.579865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.579849 5515 cpu_manager.go:414] "containerMap: removing container" podUID="755120a9-0431-406c-8f80-1f24374564cb" containerName="payload-processing" Oct 30 13:35:39.579948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.579866 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755120a9-0431-406c-8f80-1f24374564cb" containerName="payload-processing" Oct 30 13:35:39.579948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.579888 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" containerName="payload-processing" Oct 30 13:35:39.579948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.579893 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" containerName="payload-processing" Oct 30 13:35:39.580068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.580044 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="755120a9-0431-406c-8f80-1f24374564cb" containerName="payload-processing" Oct 30 13:35:39.580068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.580063 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42d2f44-6c24-4dc6-bf69-6f2049f34a0c" containerName="payload-processing" Oct 30 13:35:39.584762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.584738 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:39.606826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.606798 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx"] Oct 30 13:35:39.614043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.614017 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9a93bd\"" Oct 30 13:35:39.614043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.614024 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9a93bd-dockercfg-pvkr8\"" Oct 30 13:35:39.684188 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.684146 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:39.735073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.734990 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx\" (UID: \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:39.735251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.735156 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42hsd\" (UniqueName: \"kubernetes.io/projected/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-kube-api-access-42hsd\") pod \"payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx\" (UID: \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:39.842646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.842603 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42hsd\" (UniqueName: \"kubernetes.io/projected/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-kube-api-access-42hsd\") pod \"payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx\" (UID: \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:39.843237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.843209 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx\" (UID: \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:39.846166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.844594 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx\" (UID: \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:39.908495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.908450 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42hsd\" (UniqueName: \"kubernetes.io/projected/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-kube-api-access-42hsd\") pod \"payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx\" (UID: \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:39.914828 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:39.914790 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:40.202121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:40.202092 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:40.515628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:40.515598 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx"] Oct 30 13:35:40.516710 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:35:40.516689 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a8fa96_a0d6_4a71_b7cf_75c02aab471b.slice/crio-05ca6806726a7f4c70f714721263d947913ce035e5b67edb0a0ca0d97c6ed36f WatchSource:0}: Error finding container 05ca6806726a7f4c70f714721263d947913ce035e5b67edb0a0ca0d97c6ed36f: Status 404 returned error can't find the container with id 05ca6806726a7f4c70f714721263d947913ce035e5b67edb0a0ca0d97c6ed36f Oct 30 13:35:40.725782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:40.725751 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" event={"ID":"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b","Type":"ContainerStarted","Data":"c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8"} Oct 30 13:35:40.725965 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:40.725811 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" event={"ID":"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b","Type":"ContainerStarted","Data":"05ca6806726a7f4c70f714721263d947913ce035e5b67edb0a0ca0d97c6ed36f"} Oct 30 13:35:40.725965 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:40.725842 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:40.769141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:40.769040 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" podStartSLOduration=1.76902239 podStartE2EDuration="1.76902239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:40.766295306 +0000 UTC m=+3758.893096202" watchObservedRunningTime="2026-10-30 13:35:40.76902239 +0000 UTC m=+3758.895823288" Oct 30 13:35:41.058297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:41.058195 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:35:41.432002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:41.431954 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:41.747293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:41.747190 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:44.795235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:44.795188 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:45.663500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:45.663464 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:47.459817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:47.459782 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:48.904446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:48.904406 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:49.146451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:49.146418 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:50.920694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:50.920658 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:51.747047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:51.747016 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:35:52.715695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:52.715658 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:54.422807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:54.422770 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:56.125593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:56.125555 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:57.811038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:57.811005 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:35:58.990649 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:35:58.990612 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:35:59.490722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:35:59.490688 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:01.334618 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:01.334583 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:03.273881 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:03.273828 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:05.168031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:05.167972 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:06.208737 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.208695 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:06.428156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.428124 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:06.592373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.592300 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr"] Oct 30 13:36:06.597321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.597301 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.630442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.630410 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2d2a0b-openshift-default-dockercfg-wcwc2\"" Oct 30 13:36:06.632809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.632776 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr"] Oct 30 13:36:06.709895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.709844 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:06.719352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.719308 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18929d-bc55-4411-9648-cd8d0db04993-istiod-ca-cert\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.719776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.719750 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-credential-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.719923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.719903 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.719997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.719962 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-istio-token\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.720072 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.720012 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-envoy\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.720133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.720077 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vdj\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-kube-api-access-76vdj\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.720189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.720136 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18929d-bc55-4411-9648-cd8d0db04993-istio-podinfo\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.720323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.720303 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-certs\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.720473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.720454 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-data\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822117 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-envoy\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822193 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76vdj\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-kube-api-access-76vdj\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822252 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18929d-bc55-4411-9648-cd8d0db04993-istio-podinfo\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822426 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-certs\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822456 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-data\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822541 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18929d-bc55-4411-9648-cd8d0db04993-istiod-ca-cert\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822608 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-credential-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822685 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822738 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-istio-token\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822792 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-certs\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822849 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-data\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.822998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.822978 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-credential-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.823082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.823063 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-socket\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.823440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.823416 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18929d-bc55-4411-9648-cd8d0db04993-istiod-ca-cert\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.824751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.824731 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-envoy\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.825005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.824986 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18929d-bc55-4411-9648-cd8d0db04993-istio-podinfo\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.854336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.854261 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vdj\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-kube-api-access-76vdj\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.858866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.858842 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-istio-token\") pod \"e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:06.913361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:06.913322 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:07.053923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:07.053880 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:07.293729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:07.293672 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr"] Oct 30 13:36:07.310800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:07.310744 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:07.311940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:07.311898 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:07.312077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:07.311970 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:07.312077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:07.311998 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:36:07.648711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:07.648669 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:08.048900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:08.048803 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" event={"ID":"2f18929d-bc55-4411-9648-cd8d0db04993","Type":"ContainerStarted","Data":"d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526"} Oct 30 13:36:08.048900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:08.048861 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" event={"ID":"2f18929d-bc55-4411-9648-cd8d0db04993","Type":"ContainerStarted","Data":"8345a2444c80a267ce25abedbe9a723ff14fdcdf332ec8b2b757ec1da71fed27"} Oct 30 13:36:08.121132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:08.121057 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" podStartSLOduration=2.121035223 podStartE2EDuration="2.121035223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:08.120803698 +0000 UTC m=+3786.247604598" watchObservedRunningTime="2026-10-30 13:36:08.121035223 +0000 UTC m=+3786.247836122" Oct 30 13:36:08.835611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:08.835566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:08.915295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:08.914659 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:08.919629 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:08.919599 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 13:36:08.919934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:08.919886 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 13:36:09.877694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:09.877659 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k"] Oct 30 13:36:09.883048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:09.883026 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:09.926440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:09.926397 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 13:36:09.926633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:09.926457 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 13:36:09.932069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:09.932033 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k"] Oct 30 13:36:09.939834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:09.939808 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d41320\"/\"openshift-service-ca.crt\"" Oct 30 13:36:09.940007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:09.939828 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d41320\"/\"test-model-d41320-kserve-self-signed-certs\"" Oct 30 13:36:09.940105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:09.939887 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d41320\"/\"kube-root-ca.crt\"" Oct 30 13:36:09.940105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:09.939890 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d41320\"/\"default-dockercfg-8pf2k\"" Oct 30 13:36:10.028012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.027979 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-dshm\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.028012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.028017 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tmp-dir\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.028236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.028047 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpd8\" (UniqueName: \"kubernetes.io/projected/3c993571-01f4-4d99-81a0-1c7428a5e4b3-kube-api-access-mhpd8\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.028236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.028155 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-model-cache\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.028236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.028208 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-home\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.028390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.028361 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tls-certs\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.059412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.059374 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:10.131137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.130226 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-dshm\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.131137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.130312 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tmp-dir\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.131137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.130348 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpd8\" (UniqueName: \"kubernetes.io/projected/3c993571-01f4-4d99-81a0-1c7428a5e4b3-kube-api-access-mhpd8\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.131137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.130397 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-model-cache\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.131137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.130456 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-home\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.131137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.130563 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tls-certs\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.142129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.132728 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tmp-dir\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.142129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.135402 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-home\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.142129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.135564 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-model-cache\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.155647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.155522 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tls-certs\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.166404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.165225 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-dshm\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.175019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.174042 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpd8\" (UniqueName: \"kubernetes.io/projected/3c993571-01f4-4d99-81a0-1c7428a5e4b3-kube-api-access-mhpd8\") pod \"test-model-d41320-kserve-7dd65f975-nq42k\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.238936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.237610 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:10.647684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.646173 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:10.649533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.649322 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k"] Oct 30 13:36:10.916153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.915666 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 13:36:10.916153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:10.915725 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 13:36:11.076499 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:11.076457 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" event={"ID":"3c993571-01f4-4d99-81a0-1c7428a5e4b3","Type":"ContainerStarted","Data":"ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f"} Oct 30 13:36:11.076499 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:11.076504 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" event={"ID":"3c993571-01f4-4d99-81a0-1c7428a5e4b3","Type":"ContainerStarted","Data":"5915de981aa94b9bbcfda783b492a0aee94233a1b625b1300e9257dae95b770e"} Oct 30 13:36:11.124473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:11.124418 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" podStartSLOduration=2.124404536 podStartE2EDuration="2.124404536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:11.120580143 +0000 UTC m=+3789.247381041" watchObservedRunningTime="2026-10-30 13:36:11.124404536 +0000 UTC m=+3789.251205434" Oct 30 13:36:11.246850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:11.241846 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:11.778798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:11.778770 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:11.917652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:11.917616 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 13:36:11.918211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:11.918178 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 13:36:12.021345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:12.020569 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:12.528610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:12.528539 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:12.920293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:12.916614 5515 patch_prober.go:29] interesting pod/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 13:36:12.920293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:12.916675 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 13:36:13.516776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.516737 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:13.782416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.782325 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq"] Oct 30 13:36:13.841092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.812558 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:13.879017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.878535 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq"] Oct 30 13:36:13.879017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.878660 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:13.896317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.896262 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7b7407\"" Oct 30 13:36:13.914246 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.913833 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7b7407-dockercfg-kr8m9\"" Oct 30 13:36:13.960498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.960442 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:13.964030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.964006 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:13.965250 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.965226 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:36:13.973874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.973845 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbqw\" (UniqueName: \"kubernetes.io/projected/27c5eebc-f9da-442c-81b9-831d45fe4c01-kube-api-access-4jbqw\") pod \"payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq\" (UID: \"27c5eebc-f9da-442c-81b9-831d45fe4c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:13.974120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:13.974101 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c5eebc-f9da-442c-81b9-831d45fe4c01-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq\" (UID: \"27c5eebc-f9da-442c-81b9-831d45fe4c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:14.075389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:14.075292 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbqw\" (UniqueName: \"kubernetes.io/projected/27c5eebc-f9da-442c-81b9-831d45fe4c01-kube-api-access-4jbqw\") pod \"payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq\" (UID: \"27c5eebc-f9da-442c-81b9-831d45fe4c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:14.075772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:14.075747 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c5eebc-f9da-442c-81b9-831d45fe4c01-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq\" (UID: \"27c5eebc-f9da-442c-81b9-831d45fe4c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:14.077304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:14.077281 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c5eebc-f9da-442c-81b9-831d45fe4c01-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq\" (UID: \"27c5eebc-f9da-442c-81b9-831d45fe4c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:14.100327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:14.100299 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbqw\" (UniqueName: \"kubernetes.io/projected/27c5eebc-f9da-442c-81b9-831d45fe4c01-kube-api-access-4jbqw\") pod \"payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq\" (UID: \"27c5eebc-f9da-442c-81b9-831d45fe4c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:14.231970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:14.231940 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:14.435710 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:14.435678 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:14.556934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:14.556909 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq"] Oct 30 13:36:14.557063 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:36:14.557021 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c5eebc_f9da_442c_81b9_831d45fe4c01.slice/crio-e04c03d1fe37d5e5f4446a5f7dc5dbb046997df7346df538a325c8b54dc0db9b WatchSource:0}: Error finding container e04c03d1fe37d5e5f4446a5f7dc5dbb046997df7346df538a325c8b54dc0db9b: Status 404 returned error can't find the container with id e04c03d1fe37d5e5f4446a5f7dc5dbb046997df7346df538a325c8b54dc0db9b Oct 30 13:36:14.663792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:14.663758 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:15.111515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:15.111481 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" event={"ID":"27c5eebc-f9da-442c-81b9-831d45fe4c01","Type":"ContainerStarted","Data":"7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad"} Oct 30 13:36:15.111945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:15.111541 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" event={"ID":"27c5eebc-f9da-442c-81b9-831d45fe4c01","Type":"ContainerStarted","Data":"e04c03d1fe37d5e5f4446a5f7dc5dbb046997df7346df538a325c8b54dc0db9b"} Oct 30 13:36:15.111945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:15.111773 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:15.152222 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:15.152163 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" podStartSLOduration=2.152147777 podStartE2EDuration="2.152147777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:15.15170697 +0000 UTC m=+3793.278507880" watchObservedRunningTime="2026-10-30 13:36:15.152147777 +0000 UTC m=+3793.278948677" Oct 30 13:36:16.311674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:16.311639 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:18.105728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:18.105688 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:19.923704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:19.923671 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:20.238396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:20.238309 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:20.252438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:20.252411 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:21.170866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:21.170838 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:36:21.741557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:21.741526 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:23.389915 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:23.389879 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:25.292362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:25.292330 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:26.123010 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:26.122981 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:36:26.896999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:26.896965 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:27.258132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:27.258052 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:28.880081 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:28.873873 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:28.993629 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:36:28.993589 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:36:30.527777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:30.527743 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:32.185950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:32.185896 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:32.488685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:32.488588 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:33.887294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:33.886076 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:34.137125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:34.137084 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:35.727816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:35.727777 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:37.131616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.131585 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8"] Oct 30 13:36:37.151589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.151559 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.165741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.161624 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8"] Oct 30 13:36:37.185921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.185885 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-09ad1e-serving-cert\"" Oct 30 13:36:37.258733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.258697 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.259007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.258983 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36e1b90-f130-4b8d-be62-197d45ef6bc5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.259057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.259021 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xs4x\" (UniqueName: \"kubernetes.io/projected/b36e1b90-f130-4b8d-be62-197d45ef6bc5-kube-api-access-4xs4x\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.259139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.259123 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-metrics-certs\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.362180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.362136 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36e1b90-f130-4b8d-be62-197d45ef6bc5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.362467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.362197 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xs4x\" (UniqueName: \"kubernetes.io/projected/b36e1b90-f130-4b8d-be62-197d45ef6bc5-kube-api-access-4xs4x\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.362467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.362286 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-metrics-certs\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.362467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.362347 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.362690 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:36:37.362618 5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-serving-cert: secret "maas-api-e2e-inf-a-09ad1e-serving-cert" not found Oct 30 13:36:37.362751 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:36:37.362731 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls podName:b36e1b90-f130-4b8d-be62-197d45ef6bc5 nodeName:}" failed. No retries permitted until 2026-10-30 13:36:37.862707388 +0000 UTC m=+3815.989508287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls") pod "maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" (UID: "b36e1b90-f130-4b8d-be62-197d45ef6bc5") : secret "maas-api-e2e-inf-a-09ad1e-serving-cert" not found Oct 30 13:36:37.364345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.363833 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36e1b90-f130-4b8d-be62-197d45ef6bc5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.367498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.366916 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-metrics-certs\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.421573 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.421536 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xs4x\" (UniqueName: \"kubernetes.io/projected/b36e1b90-f130-4b8d-be62-197d45ef6bc5-kube-api-access-4xs4x\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.422838 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.422804 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:37.659102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.659072 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:37.868533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.868433 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.871069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.871040 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:37.908708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:37.908669 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:38.070291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:38.070229 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:38.422341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:38.421322 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8"] Oct 30 13:36:38.495449 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:36:38.495418 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36e1b90_f130_4b8d_be62_197d45ef6bc5.slice/crio-413ee5333f80ba8b65d8040d99775425fa7b04820d39834f2743c917d2e2feba WatchSource:0}: Error finding container 413ee5333f80ba8b65d8040d99775425fa7b04820d39834f2743c917d2e2feba: Status 404 returned error can't find the container with id 413ee5333f80ba8b65d8040d99775425fa7b04820d39834f2743c917d2e2feba Oct 30 13:36:39.311751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:39.311712 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" event={"ID":"b36e1b90-f130-4b8d-be62-197d45ef6bc5","Type":"ContainerStarted","Data":"660f1ccaec9e77437b7b806157894648ccb46b751c357e8f8114a17e3f0f850b"} Oct 30 13:36:39.311981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:39.311780 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" event={"ID":"b36e1b90-f130-4b8d-be62-197d45ef6bc5","Type":"ContainerStarted","Data":"413ee5333f80ba8b65d8040d99775425fa7b04820d39834f2743c917d2e2feba"} Oct 30 13:36:39.311981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:39.311817 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:39.338135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:39.338098 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:39.357869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:39.357819 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" podStartSLOduration=2.009359991 podStartE2EDuration="2.357803538s" totalImagesPullingTime="348.443547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:39.35566268 +0000 UTC m=+3817.482463577" watchObservedRunningTime="2026-10-30 13:36:39.357803538 +0000 UTC m=+3817.484604464" Oct 30 13:36:41.294144 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:41.294113 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:43.020706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:43.020676 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:44.945448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:44.945410 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:45.329949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:45.329874 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:36:46.631687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:46.631650 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:48.240856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:48.240826 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:50.015480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:50.015450 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:51.879656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:51.879619 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:53.656566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:53.656533 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:55.604618 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:55.604580 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:57.282700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:57.282662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:58.937165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:36:58.937135 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:36:58.989749 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:36:58.989717 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:37:00.933594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:00.933558 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:02.351672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:02.351643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:04.169808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:04.169772 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:05.651925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:05.651886 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:37:06.147295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:06.143859 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:08.060639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:08.060602 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:09.823168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:09.823131 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j"] Oct 30 13:37:09.828920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:09.828494 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:09.859612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:09.859579 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7b7407-serving-cert\"" Oct 30 13:37:09.859872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:09.859847 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j"] Oct 30 13:37:09.920709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:09.920669 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c460355f-0748-412a-a6f6-eb0a0548c441-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:09.920894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:09.920738 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72tgx\" (UniqueName: \"kubernetes.io/projected/c460355f-0748-412a-a6f6-eb0a0548c441-kube-api-access-72tgx\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:09.920894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:09.920836 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-metrics-certs\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:09.921056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:09.921036 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.027102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.027066 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c460355f-0748-412a-a6f6-eb0a0548c441-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.027425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.027404 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72tgx\" (UniqueName: \"kubernetes.io/projected/c460355f-0748-412a-a6f6-eb0a0548c441-kube-api-access-72tgx\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.027572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.027555 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-metrics-certs\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.027816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.027802 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.028441 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:37:10.028415 5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-serving-cert: secret "maas-api-e2e-inf-b-7b7407-serving-cert" not found Oct 30 13:37:10.028692 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:37:10.028679 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls podName:c460355f-0748-412a-a6f6-eb0a0548c441 nodeName:}" failed. No retries permitted until 2026-10-30 13:37:10.528655104 +0000 UTC m=+3848.655455998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls") pod "maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" (UID: "c460355f-0748-412a-a6f6-eb0a0548c441") : secret "maas-api-e2e-inf-b-7b7407-serving-cert" not found Oct 30 13:37:10.029789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.029767 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c460355f-0748-412a-a6f6-eb0a0548c441-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.035572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.035331 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-metrics-certs\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.064553 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.064521 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72tgx\" (UniqueName: \"kubernetes.io/projected/c460355f-0748-412a-a6f6-eb0a0548c441-kube-api-access-72tgx\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.133586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.133561 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:10.525849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.525767 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:10.535523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.535492 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.538077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.538054 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls\") pod \"maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:10.748187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:10.748151 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:11.121295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:11.111840 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j"] Oct 30 13:37:11.579090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:11.579057 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" event={"ID":"c460355f-0748-412a-a6f6-eb0a0548c441","Type":"ContainerStarted","Data":"1d3f07dc28a9b7db7bb3664809dc8f0dc7d73098a0216462e938f9cdaa8ca37a"} Oct 30 13:37:12.028876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:12.028843 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:12.588478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:12.588441 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" event={"ID":"c460355f-0748-412a-a6f6-eb0a0548c441","Type":"ContainerStarted","Data":"14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642"} Oct 30 13:37:12.588896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:12.588534 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:12.633406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:12.633350 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" podStartSLOduration=3.231549429 podStartE2EDuration="3.633334575s" totalImagesPullingTime="401.785146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:12.631452838 +0000 UTC m=+3850.758253736" watchObservedRunningTime="2026-10-30 13:37:12.633334575 +0000 UTC m=+3850.760135473" Oct 30 13:37:13.859971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:13.859913 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:14.658653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:14.658614 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:15.491968 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:15.491927 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:17.304749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:17.304713 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:18.602952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:18.602923 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:37:18.970841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:18.970803 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:20.643934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:20.643902 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:22.422682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:22.422637 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:24.111495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.111464 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:24.813749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.813717 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll"] Oct 30 13:37:24.819113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.819091 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.834452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.834427 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ba646b-openshift-default-dockercfg-wjbl8\"" Oct 30 13:37:24.852211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.852178 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll"] Oct 30 13:37:24.868007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.867970 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-envoy\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.868170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.868054 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-data\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.868170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.868088 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.868170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.868134 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-token\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.868354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.868252 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istiod-ca-cert\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.868354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.868340 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-certs\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.868437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.868396 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-credential-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.868478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.868440 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqlzt\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-kube-api-access-kqlzt\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.868478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.868468 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-podinfo\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.969919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.969882 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-envoy\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.969942 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-data\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970059 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970114 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-token\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970239 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istiod-ca-cert\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970316 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-data\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970354 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-certs\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970410 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-credential-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970464 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqlzt\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-kube-api-access-kqlzt\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970467 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970505 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-podinfo\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970696 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-certs\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.970983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.970963 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-credential-socket\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.971067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.971042 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istiod-ca-cert\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.972457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.972434 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-envoy\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.973017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.972999 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-podinfo\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.994034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.994009 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-token\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:24.994252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:24.994231 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqlzt\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-kube-api-access-kqlzt\") pod \"e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:25.134817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:25.134783 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:25.462052 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:37:25.462021 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf0cda3_bd5d_4ab1_9695_2e435758cc6e.slice/crio-0f4b3f8bf6648d5ec1ce27b1126e828166c83ad876a0283e2150aee106ce0728 WatchSource:0}: Error finding container 0f4b3f8bf6648d5ec1ce27b1126e828166c83ad876a0283e2150aee106ce0728: Status 404 returned error can't find the container with id 0f4b3f8bf6648d5ec1ce27b1126e828166c83ad876a0283e2150aee106ce0728 Oct 30 13:37:25.463552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:25.463527 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll"] Oct 30 13:37:25.465526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:25.465492 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:25.465640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:25.465575 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:25.465640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:25.465622 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:25.696835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:25.696791 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" event={"ID":"abf0cda3-bd5d-4ab1-9695-2e435758cc6e","Type":"ContainerStarted","Data":"426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d"} Oct 30 13:37:25.697037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:25.696855 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" event={"ID":"abf0cda3-bd5d-4ab1-9695-2e435758cc6e","Type":"ContainerStarted","Data":"0f4b3f8bf6648d5ec1ce27b1126e828166c83ad876a0283e2150aee106ce0728"} Oct 30 13:37:25.744614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:25.744463 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" podStartSLOduration=1.744441948 podStartE2EDuration="1.744441948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:25.742532941 +0000 UTC m=+3863.869333856" watchObservedRunningTime="2026-10-30 13:37:25.744441948 +0000 UTC m=+3863.871242847" Oct 30 13:37:25.798407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:25.798372 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:26.135457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:26.135427 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:26.137135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:26.137112 5515 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 13:37:26.137251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:26.137166 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 13:37:27.135466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:27.135431 5515 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 13:37:27.135887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:27.135487 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 13:37:27.542400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:27.542314 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:28.135977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:28.135940 5515 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 13:37:28.138496 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:28.136013 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 13:37:29.137294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:29.135530 5515 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 13:37:29.137294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:29.135595 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 13:37:29.142936 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:37:29.142907 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:37:29.307756 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:29.307719 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:30.141336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:30.141298 5515 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 13:37:30.141999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:30.141961 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 13:37:30.192527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:30.192494 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:30.715291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:30.714304 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:31.135307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:31.135252 5515 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 13:37:31.135518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:31.135359 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 13:37:31.179100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:31.179071 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:31.418880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:31.418788 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:32.135611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:32.135577 5515 patch_prober.go:29] interesting pod/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 13:37:32.135799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:32.135636 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 13:37:32.990883 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:32.990852 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:33.240456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:33.240422 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:33.240849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:33.240827 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:33.241778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:33.241726 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:37:34.671332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:34.671290 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:36.598561 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:36.598527 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:36.904373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:36.904337 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:37.130205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:37.130170 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:38.599769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:38.599736 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:39.005719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:39.005668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:40.599668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:40.599636 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:40.853896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:40.853815 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:42.534995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:42.534870 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:42.797338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:42.790167 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:43.143086 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:43.143025 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:43.651298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:43.650053 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:43.886690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:43.886654 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:44.382810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:44.382775 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:46.238697 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:46.238664 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:48.138119 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.138088 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:48.727353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.727320 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc"] Oct 30 13:37:48.728846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.728817 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:48.733896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.733872 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.780903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.780852 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc"] Oct 30 13:37:48.823357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.823325 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-envoy\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.823528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.823369 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istiod-ca-cert\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.823528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.823394 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-credential-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.823528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.823496 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-certs\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.823668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.823558 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-podinfo\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.823734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.823682 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnsdn\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-kube-api-access-hnsdn\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.823793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.823750 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-token\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.823846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.823825 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.823956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.823941 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-data\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.825884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.825862 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9a93bd-openshift-default-dockercfg-mgbm5\"" Oct 30 13:37:48.925890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.925861 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-data\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.925945 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-envoy\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.925969 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istiod-ca-cert\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.925988 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-credential-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926010 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-certs\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926052 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-podinfo\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926135 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnsdn\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-kube-api-access-hnsdn\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926286 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-data\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926300 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-token\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926377 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926524 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-credential-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926571 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-certs\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926754 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926721 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.926795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.926783 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istiod-ca-cert\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.928682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.928652 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-envoy\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.928819 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.928761 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-podinfo\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.949814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.949787 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-token\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.949934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.949899 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnsdn\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-kube-api-access-hnsdn\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:48.956394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:48.956363 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:49.086467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:49.086390 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:49.411838 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:49.411813 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc"] Oct 30 13:37:49.412214 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:37:49.412192 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c8d1c1_6e77_4ef4_8372_142e32c0004b.slice/crio-e521b8444deab1c2e7372b81a9f453e5cf13e37a5e6da7ffa3a8a8e3622345e7 WatchSource:0}: Error finding container e521b8444deab1c2e7372b81a9f453e5cf13e37a5e6da7ffa3a8a8e3622345e7: Status 404 returned error can't find the container with id e521b8444deab1c2e7372b81a9f453e5cf13e37a5e6da7ffa3a8a8e3622345e7 Oct 30 13:37:49.415310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:49.415247 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:49.415408 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:49.415332 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:49.415408 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:49.415364 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:37:49.953316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:49.953259 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:50.004356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:50.004322 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" event={"ID":"d9c8d1c1-6e77-4ef4-8372-142e32c0004b","Type":"ContainerStarted","Data":"6c36bada41af4544226d789a45f29e84d2d36dda26bb510c4ec7eae91bf8e286"} Oct 30 13:37:50.004526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:50.004383 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" event={"ID":"d9c8d1c1-6e77-4ef4-8372-142e32c0004b","Type":"ContainerStarted","Data":"e521b8444deab1c2e7372b81a9f453e5cf13e37a5e6da7ffa3a8a8e3622345e7"} Oct 30 13:37:50.052303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:50.052218 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" podStartSLOduration=2.052205152 podStartE2EDuration="2.052205152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:50.051939955 +0000 UTC m=+3888.178740854" watchObservedRunningTime="2026-10-30 13:37:50.052205152 +0000 UTC m=+3888.179006050" Oct 30 13:37:50.086764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:50.086736 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:50.088604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:50.088573 5515 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 13:37:50.088722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:50.088629 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 13:37:50.478869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:50.478827 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:50.849006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:50.848918 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:51.087675 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:51.087647 5515 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 13:37:51.087832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:51.087690 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 13:37:51.877074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:51.877034 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:52.086856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:52.086823 5515 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 13:37:52.087032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:52.086873 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 13:37:53.087127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:53.087093 5515 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 13:37:53.087550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:53.087157 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 13:37:53.642758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:53.642721 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:54.087078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:54.086999 5515 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 13:37:54.087078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:54.087046 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 13:37:55.087110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:55.087076 5515 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 13:37:55.087321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:55.087125 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 13:37:55.323160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:55.323121 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:56.086946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:56.086912 5515 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 13:37:56.087113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:56.086991 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 13:37:57.025729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:57.025689 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:57.092848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:57.092818 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:57.093256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:57.093230 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:57.093930 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:57.093911 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:37:58.781425 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:37:58.781395 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:37:58.991012 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:37:58.990973 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:38:00.393894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:00.393829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:02.320722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:02.320687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:04.129396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:04.129362 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:06.019474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:06.019441 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:07.645991 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:07.645958 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:08.911719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:08.911684 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:09.523189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:09.523157 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:11.335123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:11.335079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:13.240638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:13.240605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:14.804893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:14.804861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:15.031926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:15.031893 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:15.379201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:15.379161 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:16.939346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:16.939297 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:17.188459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:17.188425 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:18.717514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:18.717479 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:20.495654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:20.495621 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:22.208076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:22.208036 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:22.571613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:22.571531 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:23.986739 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:23.986707 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:25.618807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:25.618768 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:26.984300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:26.984255 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:38:27.257771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:27.257692 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:29.040112 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:38:29.040070 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:38:29.118071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:29.118038 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:30.812637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:30.812603 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:32.530118 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:32.530079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:34.312485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:34.312414 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:36.086616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:36.086581 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:37.597598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.597558 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll"] Oct 30 13:38:37.602437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.602414 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.617745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.616857 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrx2s\" (UniqueName: \"kubernetes.io/projected/a50c9b9b-2449-49e4-ab44-f03c33a67950-kube-api-access-qrx2s\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.617745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.617205 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9b9b-2449-49e4-ab44-f03c33a67950-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.617745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.617235 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-metrics-certs\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.617745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.617470 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.636609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.636575 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll"] Oct 30 13:38:37.640074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.640039 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f97234-serving-cert\"" Oct 30 13:38:37.721004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.720959 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9b9b-2449-49e4-ab44-f03c33a67950-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.722063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.721259 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-metrics-certs\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.722063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.721506 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.722063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.721586 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrx2s\" (UniqueName: \"kubernetes.io/projected/a50c9b9b-2449-49e4-ab44-f03c33a67950-kube-api-access-qrx2s\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.722063 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:38:37.721745 5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-serving-cert: secret "maas-api-e2e-inf-b-f97234-serving-cert" not found Oct 30 13:38:37.722063 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:38:37.721824 5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls podName:a50c9b9b-2449-49e4-ab44-f03c33a67950 nodeName:}" failed. No retries permitted until 2026-10-30 13:38:38.221804696 +0000 UTC m=+3936.348605572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls") pod "maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" (UID: "a50c9b9b-2449-49e4-ab44-f03c33a67950") : secret "maas-api-e2e-inf-b-f97234-serving-cert" not found Oct 30 13:38:37.722063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.721882 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9b9b-2449-49e4-ab44-f03c33a67950-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.724252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.724225 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-metrics-certs\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.746296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.746248 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrx2s\" (UniqueName: \"kubernetes.io/projected/a50c9b9b-2449-49e4-ab44-f03c33a67950-kube-api-access-qrx2s\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:37.789978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:37.789928 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:38.012321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:38.012285 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:38.230284 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:38.230242 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:38.232701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:38.232676 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls\") pod \"maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:38.520258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:38.520221 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:38.927936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:38.927116 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll"] Oct 30 13:38:39.443589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:39.443556 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" event={"ID":"a50c9b9b-2449-49e4-ab44-f03c33a67950","Type":"ContainerStarted","Data":"a3003cb8ac63c0d5b344f80b997497e268d2eccee07085c39f80cf285ec1403a"} Oct 30 13:38:39.658262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:39.658098 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:40.453587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:40.453553 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" event={"ID":"a50c9b9b-2449-49e4-ab44-f03c33a67950","Type":"ContainerStarted","Data":"fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82"} Oct 30 13:38:40.453998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:40.453706 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:40.502223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:40.502167 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" podStartSLOduration=3.021901246 podStartE2EDuration="3.502152904s" totalImagesPullingTime="480.251658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:40.497245632 +0000 UTC m=+3938.624046531" watchObservedRunningTime="2026-10-30 13:38:40.502152904 +0000 UTC m=+3938.628953830" Oct 30 13:38:41.403633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:41.403593 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:41.984730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:41.984687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:43.236750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:43.236712 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:44.963988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:44.963953 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:46.346473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:46.346432 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:46.468970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:46.468944 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:38:46.583888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:46.583850 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:46.807983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:46.807954 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:48.428799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:48.428755 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:50.142648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:50.142618 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:50.362765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:50.362725 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:51.920676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:51.920648 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:53.761605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:53.761567 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:55.571954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:55.571923 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:57.342959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:57.342926 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:38:59.008852 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:38:59.008806 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:38:59.026954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:38:59.026925 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:00.694657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:00.694619 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:02.436104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:02.436070 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:04.212003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:04.211965 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:05.927181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:05.927144 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:07.533048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:07.533015 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:09.145371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:09.145337 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:10.990142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:10.990110 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:12.688916 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:12.688829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:14.394741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:14.394702 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:14.908457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:14.908421 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:16.044157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:16.044126 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:17.796058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:17.795994 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:19.529675 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:19.529641 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:21.282819 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:21.282776 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:22.978211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:22.978173 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:24.763577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:24.763540 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:26.400685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:26.400649 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:28.024852 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:28.024818 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:28.996734 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:39:28.996643 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:39:29.714443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:29.714407 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:31.629477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:31.629438 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:33.287124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:33.287076 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:34.448606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:34.448559 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:34.668830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:34.668764 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:34.910977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:34.910935 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:36.658150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:36.658098 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:38.038345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:38.038305 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:39:38.327022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:38.326935 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:40.120803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:40.120768 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:41.960759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:41.960726 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:43.931923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:43.931783 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:45.286665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:45.286634 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:45.641751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:45.641710 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:47.324973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:47.324904 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:48.912227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:48.912193 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:49.176507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:49.176414 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:50.822831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:50.822797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:52.491841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:52.491805 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:54.453588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:54.453556 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:56.364928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:56.364786 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:57.017053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:57.017012 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:58.184986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:58.184942 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:39:58.989865 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:39:58.989833 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:39:59.982827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:39:59.982796 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:01.169781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.169752 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg"] Oct 30 13:40:01.175952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.175930 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.192372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.192348 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f97234-openshift-default-dockercfg-x8qzl\"" Oct 30 13:40:01.199003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.198977 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg"] Oct 30 13:40:01.199698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.199678 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.199769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.199750 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.199902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.199884 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.199957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.199910 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.199957 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.199929 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tszgn\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-kube-api-access-tszgn\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.200081 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.200045 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.200193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.200167 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.200358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.200247 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.200358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.200347 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.301511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.301470 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.301698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.301581 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.301771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.301703 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.301771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.301732 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.301771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.301761 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszgn\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-kube-api-access-tszgn\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.301936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.301894 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.301993 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.301934 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.302059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.302034 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.302120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.302087 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.302180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.302115 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.302237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.302208 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.302361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.302334 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.302483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.302332 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.302529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.302500 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.304059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.304026 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.304674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.304651 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.325669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.325646 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tszgn\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-kube-api-access-tszgn\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.325669 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.325658 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.491947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.491871 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:01.826937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.826734 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg"] Oct 30 13:40:01.831553 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:40:01.831523 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dbcfc4_c27b_4955_bfe1_2420985e5687.slice/crio-1600d1daa35c54e058c011b6d1ed3899c515c5d894b5ca3f4f7b99b88ea88a94 WatchSource:0}: Error finding container 1600d1daa35c54e058c011b6d1ed3899c515c5d894b5ca3f4f7b99b88ea88a94: Status 404 returned error can't find the container with id 1600d1daa35c54e058c011b6d1ed3899c515c5d894b5ca3f4f7b99b88ea88a94 Oct 30 13:40:01.834845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.834796 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:40:01.834944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.834917 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:40:01.834995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.834962 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:40:01.839891 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:01.839846 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:02.257450 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:02.257414 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" event={"ID":"a0dbcfc4-c27b-4955-bfe1-2420985e5687","Type":"ContainerStarted","Data":"74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255"} Oct 30 13:40:02.257863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:02.257468 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" event={"ID":"a0dbcfc4-c27b-4955-bfe1-2420985e5687","Type":"ContainerStarted","Data":"1600d1daa35c54e058c011b6d1ed3899c515c5d894b5ca3f4f7b99b88ea88a94"} Oct 30 13:40:02.323752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:02.323693 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" podStartSLOduration=1.323679909 podStartE2EDuration="1.323679909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:02.322014869 +0000 UTC m=+4020.448815768" watchObservedRunningTime="2026-10-30 13:40:02.323679909 +0000 UTC m=+4020.450480807" Oct 30 13:40:02.492467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:02.492435 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:02.494174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:02.494149 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:40:02.494320 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:02.494196 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 13:40:03.492977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:03.492945 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:40:03.493530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:03.493011 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 13:40:03.629451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:03.629405 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:04.492299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:04.492249 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:40:04.492488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:04.492317 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 13:40:05.492750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:05.492715 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:40:05.493187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:05.492778 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 13:40:05.505351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:05.505164 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:06.492321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:06.492258 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:40:06.492494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:06.492350 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 13:40:07.236333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:07.235168 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:07.492278 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:07.492181 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 13:40:07.492448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:07.492294 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 13:40:08.501129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:08.501098 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:08.501651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:08.501616 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:08.502683 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:08.502664 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:40:09.009884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:09.009836 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:10.735636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:10.735605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:12.483758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:12.483714 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:14.194249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:14.194213 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:14.999629 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:14.999595 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:15.935017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:15.934986 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:16.146924 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:16.146887 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:17.727673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:17.727632 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:19.391826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:19.391795 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:21.179477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:21.179434 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:22.275498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:22.275464 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:22.899610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:22.899573 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:24.685856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:24.685811 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:26.301791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:26.301757 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:28.019800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:28.019764 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:28.991364 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:40:28.991327 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:40:29.777866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:29.777811 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:30.560526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.560489 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6"] Oct 30 13:40:30.566256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.566234 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.607781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.607753 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-af8369\"/\"openshift-service-ca.crt\"" Oct 30 13:40:30.607960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.607928 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-af8369\"/\"kube-root-ca.crt\"" Oct 30 13:40:30.608074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.608055 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-af8369\"/\"default-dockercfg-lwbtv\"" Oct 30 13:40:30.608311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.608286 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-af8369\"/\"test-model-af8369-kserve-self-signed-certs\"" Oct 30 13:40:30.615192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.615163 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6"] Oct 30 13:40:30.680711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.680678 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh9zx\" (UniqueName: \"kubernetes.io/projected/cc50345e-fefa-4329-a72c-2248c950a603-kube-api-access-sh9zx\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.680896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.680782 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-dshm\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.680981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.680964 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-model-cache\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.681033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.681015 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-home\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.681076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.681035 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-tmp-dir\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.681197 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.681174 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc50345e-fefa-4329-a72c-2248c950a603-tls-certs\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.723120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.723087 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:30.782513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.782480 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc50345e-fefa-4329-a72c-2248c950a603-tls-certs\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.782999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.782563 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh9zx\" (UniqueName: \"kubernetes.io/projected/cc50345e-fefa-4329-a72c-2248c950a603-kube-api-access-sh9zx\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.782999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.782611 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-dshm\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.782999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.782740 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-model-cache\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.782999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.782774 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-home\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.782999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.782791 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-tmp-dir\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.783327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.783148 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-model-cache\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.783327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.783188 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-home\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.783327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.783221 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-tmp-dir\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.784876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.784853 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-dshm\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.785050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.785033 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc50345e-fefa-4329-a72c-2248c950a603-tls-certs\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.806165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.806142 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh9zx\" (UniqueName: \"kubernetes.io/projected/cc50345e-fefa-4329-a72c-2248c950a603-kube-api-access-sh9zx\") pod \"test-model-af8369-kserve-6d85cc47b5-lpvg6\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:30.897346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:30.881568 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:31.284530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.284497 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt"] Oct 30 13:40:31.307528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.305640 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6"] Oct 30 13:40:31.307528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.305770 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.327366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.327334 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt"] Oct 30 13:40:31.407776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.407742 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.407923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.407780 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.407923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.407872 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.408020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.407938 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.408104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.408080 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7252b9af-d947-446c-bf46-0d6802a57e12-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.408179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.408121 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7252b9af-d947-446c-bf46-0d6802a57e12-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.408321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.408306 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfwlb\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-kube-api-access-kfwlb\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.408381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.408342 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.408381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.408360 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510138 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7252b9af-d947-446c-bf46-0d6802a57e12-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510181 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7252b9af-d947-446c-bf46-0d6802a57e12-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510367 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfwlb\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-kube-api-access-kfwlb\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510419 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510449 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510646 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510674 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510721 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510800 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510874 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510944 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.510986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.510958 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.511121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.511088 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7252b9af-d947-446c-bf46-0d6802a57e12-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.511285 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.511244 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.512814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.512788 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.512920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.512849 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7252b9af-d947-446c-bf46-0d6802a57e12-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.534426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.534373 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.534536 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.534517 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfwlb\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-kube-api-access-kfwlb\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.556026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.555993 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" event={"ID":"cc50345e-fefa-4329-a72c-2248c950a603","Type":"ContainerStarted","Data":"ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd"} Oct 30 13:40:31.556148 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.556035 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" event={"ID":"cc50345e-fefa-4329-a72c-2248c950a603","Type":"ContainerStarted","Data":"efa64e8766964b829183c88bb23f53d534f41014ffee2f46e3f77b9e3cfdaa25"} Oct 30 13:40:31.606846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.606794 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" podStartSLOduration=1.606780957 podStartE2EDuration="1.606780957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:31.603031739 +0000 UTC m=+4049.729832639" watchObservedRunningTime="2026-10-30 13:40:31.606780957 +0000 UTC m=+4049.733581881" Oct 30 13:40:31.641367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.641333 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:31.760170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.759613 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:31.882621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.882595 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:31.964826 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.964794 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt"] Oct 30 13:40:31.965044 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:40:31.965023 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7252b9af_d947_446c_bf46_0d6802a57e12.slice/crio-9e581748aa4eddd21ae23d461b8767b3ba5069fb6ebfd73776888915eb6da162 WatchSource:0}: Error finding container 9e581748aa4eddd21ae23d461b8767b3ba5069fb6ebfd73776888915eb6da162: Status 404 returned error can't find the container with id 9e581748aa4eddd21ae23d461b8767b3ba5069fb6ebfd73776888915eb6da162 Oct 30 13:40:31.968370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.968335 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:40:31.968467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.968403 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:40:31.968467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:31.968428 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:40:32.566054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:32.566014 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" event={"ID":"7252b9af-d947-446c-bf46-0d6802a57e12","Type":"ContainerStarted","Data":"d0a1ee1142db26a0e7eef572f649ca8bb4e660ab60dd7bbe0f5dc117483616a4"} Oct 30 13:40:32.566259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:32.566080 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" event={"ID":"7252b9af-d947-446c-bf46-0d6802a57e12","Type":"ContainerStarted","Data":"9e581748aa4eddd21ae23d461b8767b3ba5069fb6ebfd73776888915eb6da162"} Oct 30 13:40:32.637102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:32.637047 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" podStartSLOduration=1.63703326 podStartE2EDuration="1.63703326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:32.635626034 +0000 UTC m=+4050.762426933" watchObservedRunningTime="2026-10-30 13:40:32.63703326 +0000 UTC m=+4050.763834157" Oct 30 13:40:32.642124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:32.642102 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:33.623291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:33.619662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:33.660291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:33.652385 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:40:33.660291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:33.652447 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:40:33.853831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:33.853800 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:34.642332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:34.642301 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:40:34.642732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:34.642350 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:40:35.627295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:35.619338 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:35.642202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:35.642169 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:40:35.642421 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:35.642223 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:40:36.641870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:36.641837 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:40:36.642057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:36.641890 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:40:37.431500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:37.426000 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:37.641719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:37.641686 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:40:37.641938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:37.641737 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:40:38.642438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:38.642400 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 13:40:38.642934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:38.642460 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 13:40:39.274239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:39.274195 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:39.648012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:39.647983 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:39.648587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:39.648564 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:39.649286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:39.649248 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:40:40.882256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:40.882224 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:40.895191 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:40.895160 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:41.104768 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:41.104732 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:41.685835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:41.685804 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:40:42.779628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:42.779578 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:44.679803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:44.679767 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:46.500208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:46.500170 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:46.992878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:46.992846 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:48.324782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:48.324749 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:50.112397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:50.112358 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:51.783708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:51.783674 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:53.826719 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:53.826637 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:55.513723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:55.513678 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:57.013168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:57.013122 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:57.233097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:57.233041 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:57.387889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:57.387852 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k"] Oct 30 13:40:57.542994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:57.542935 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:58.977233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:58.977198 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:40:58.999622 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:40:58.999587 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:40:59.824234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:40:59.824190 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:00.062878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:00.062842 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7"] Oct 30 13:41:00.063304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:00.063077 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" containerID="cri-o://475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618" gracePeriod=30 Oct 30 13:41:00.678436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:00.677724 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:02.319065 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:02.319033 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:04.108824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:04.108779 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:05.360552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.360482 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:41:05.462734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.462691 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-envoy\") pod \"42262685-7312-4ac0-8479-be793dab101c\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " Oct 30 13:41:05.462964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.462815 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-certs\") pod \"42262685-7312-4ac0-8479-be793dab101c\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " Oct 30 13:41:05.462964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.462943 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-socket\") pod \"42262685-7312-4ac0-8479-be793dab101c\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " Oct 30 13:41:05.463101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.462982 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-data\") pod \"42262685-7312-4ac0-8479-be793dab101c\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " Oct 30 13:41:05.463167 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.463084 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-certs" pod "42262685-7312-4ac0-8479-be793dab101c" (UID: "42262685-7312-4ac0-8479-be793dab101c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:05.463358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.463330 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-socket" pod "42262685-7312-4ac0-8479-be793dab101c" (UID: "42262685-7312-4ac0-8479-be793dab101c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:05.463471 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.463380 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-data" pod "42262685-7312-4ac0-8479-be793dab101c" (UID: "42262685-7312-4ac0-8479-be793dab101c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:05.463559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.463540 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-istio-token\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-istio-token\") pod \"42262685-7312-4ac0-8479-be793dab101c\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " Oct 30 13:41:05.463636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.463620 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-credential-socket\") pod \"42262685-7312-4ac0-8479-be793dab101c\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " Oct 30 13:41:05.463737 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.463719 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42262685-7312-4ac0-8479-be793dab101c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42262685-7312-4ac0-8479-be793dab101c-istio-podinfo\") pod \"42262685-7312-4ac0-8479-be793dab101c\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " Oct 30 13:41:05.463821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.463757 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-kube-api-access-zv2ct\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-kube-api-access-zv2ct\") pod \"42262685-7312-4ac0-8479-be793dab101c\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " Oct 30 13:41:05.463888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.463863 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42262685-7312-4ac0-8479-be793dab101c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42262685-7312-4ac0-8479-be793dab101c-istiod-ca-cert\") pod \"42262685-7312-4ac0-8479-be793dab101c\" (UID: \"42262685-7312-4ac0-8479-be793dab101c\") " Oct 30 13:41:05.463948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.463899 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-credential-socket" pod "42262685-7312-4ac0-8479-be793dab101c" (UID: "42262685-7312-4ac0-8479-be793dab101c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:05.464850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.464583 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42262685-7312-4ac0-8479-be793dab101c-istiod-ca-cert" pod "42262685-7312-4ac0-8479-be793dab101c" (UID: "42262685-7312-4ac0-8479-be793dab101c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:05.464850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.464753 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.464850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.464774 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42262685-7312-4ac0-8479-be793dab101c-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.464850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.464793 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.464850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.464807 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.464850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.464823 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.465544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.465516 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-envoy" pod "42262685-7312-4ac0-8479-be793dab101c" (UID: "42262685-7312-4ac0-8479-be793dab101c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:05.466259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.466217 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-istio-token" pod "42262685-7312-4ac0-8479-be793dab101c" (UID: "42262685-7312-4ac0-8479-be793dab101c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:05.466716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.466683 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-kube-api-access-zv2ct" pod "42262685-7312-4ac0-8479-be793dab101c" (UID: "42262685-7312-4ac0-8479-be793dab101c"). InnerVolumeSpecName "kube-api-access-zv2ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:05.467124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.467103 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42262685-7312-4ac0-8479-be793dab101c-istio-podinfo" pod "42262685-7312-4ac0-8479-be793dab101c" (UID: "42262685-7312-4ac0-8479-be793dab101c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:05.566363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.566326 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42262685-7312-4ac0-8479-be793dab101c-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.566363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.566358 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv2ct\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-kube-api-access-zv2ct\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.566363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.566369 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42262685-7312-4ac0-8479-be793dab101c-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.566613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.566379 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42262685-7312-4ac0-8479-be793dab101c-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:05.895696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.895643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:05.936836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.936801 5515 generic.go:356] "Generic (PLEG): container finished" podID="42262685-7312-4ac0-8479-be793dab101c" containerID="475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618" exitCode=0 Oct 30 13:41:05.937043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.936868 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" Oct 30 13:41:05.937043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.936885 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" event={"ID":"42262685-7312-4ac0-8479-be793dab101c","Type":"ContainerDied","Data":"475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618"} Oct 30 13:41:05.937043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.936944 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7" event={"ID":"42262685-7312-4ac0-8479-be793dab101c","Type":"ContainerDied","Data":"d549fb81f94e8a4ca4c2538d65108a7283e6340563f194f51ec8dc196d2dc64e"} Oct 30 13:41:05.937043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.936970 5515 scope.go:122] "RemoveContainer" containerID="475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618" Oct 30 13:41:05.949849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.949828 5515 scope.go:122] "RemoveContainer" containerID="475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618" Oct 30 13:41:05.950193 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:41:05.950173 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618\": container with ID starting with 475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618 not found: ID does not exist" containerID="475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618" Oct 30 13:41:05.950245 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:05.950204 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618"} err="failed to get container status \"475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618\": rpc error: code = NotFound desc = could not find container \"475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618\": container with ID starting with 475ad4c5b4b52365d4170425bcb2f2dc2b8a5b31207862865bb83ae8c0e08618 not found: ID does not exist" Oct 30 13:41:06.006360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:06.006333 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7"] Oct 30 13:41:06.009897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:06.009876 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7jlq7"] Oct 30 13:41:07.835915 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:07.835878 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:08.004802 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:08.004765 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42262685-7312-4ac0-8479-be793dab101c" path="/var/lib/kubelet/pods/42262685-7312-4ac0-8479-be793dab101c/volumes" Oct 30 13:41:08.095613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:08.095533 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:41:09.583478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:09.583440 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:11.382976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:11.382933 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:12.410705 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.410665 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" podUID="3c993571-01f4-4d99-81a0-1c7428a5e4b3" containerName="main" containerID="cri-o://ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f" gracePeriod=45 Oct 30 13:41:12.729338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.729310 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:41:12.845999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.845963 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tmp-dir\") pod \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " Oct 30 13:41:12.846215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.846032 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tls-certs\") pod \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " Oct 30 13:41:12.846215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.846103 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-model-cache\") pod \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " Oct 30 13:41:12.846382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.846215 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c993571-01f4-4d99-81a0-1c7428a5e4b3-kube-api-access-mhpd8\" (UniqueName: \"kubernetes.io/projected/3c993571-01f4-4d99-81a0-1c7428a5e4b3-kube-api-access-mhpd8\") pod \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " Oct 30 13:41:12.846382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.846305 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-home\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-home\") pod \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " Oct 30 13:41:12.846382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.846314 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tmp-dir" pod "3c993571-01f4-4d99-81a0-1c7428a5e4b3" (UID: "3c993571-01f4-4d99-81a0-1c7428a5e4b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:12.846382 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.846359 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-dshm\") pod \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\" (UID: \"3c993571-01f4-4d99-81a0-1c7428a5e4b3\") " Oct 30 13:41:12.846594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.846558 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-model-cache" pod "3c993571-01f4-4d99-81a0-1c7428a5e4b3" (UID: "3c993571-01f4-4d99-81a0-1c7428a5e4b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:12.847073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.847049 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.847073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.847061 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-home" pod "3c993571-01f4-4d99-81a0-1c7428a5e4b3" (UID: "3c993571-01f4-4d99-81a0-1c7428a5e4b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:12.847073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.847074 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.848987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.848959 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tls-certs" pod "3c993571-01f4-4d99-81a0-1c7428a5e4b3" (UID: "3c993571-01f4-4d99-81a0-1c7428a5e4b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:12.849119 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.849049 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-dshm" pod "3c993571-01f4-4d99-81a0-1c7428a5e4b3" (UID: "3c993571-01f4-4d99-81a0-1c7428a5e4b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:12.849583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.849562 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c993571-01f4-4d99-81a0-1c7428a5e4b3-kube-api-access-mhpd8" pod "3c993571-01f4-4d99-81a0-1c7428a5e4b3" (UID: "3c993571-01f4-4d99-81a0-1c7428a5e4b3"). InnerVolumeSpecName "kube-api-access-mhpd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:12.948887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.948794 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c993571-01f4-4d99-81a0-1c7428a5e4b3-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.948887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.948826 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhpd8\" (UniqueName: \"kubernetes.io/projected/3c993571-01f4-4d99-81a0-1c7428a5e4b3-kube-api-access-mhpd8\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.948887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.948839 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:12.948887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:12.948853 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c993571-01f4-4d99-81a0-1c7428a5e4b3-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:13.033807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.033765 5515 generic.go:356] "Generic (PLEG): container finished" podID="3c993571-01f4-4d99-81a0-1c7428a5e4b3" containerID="ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f" exitCode=0 Oct 30 13:41:13.033980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.033827 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" Oct 30 13:41:13.033980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.033851 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" event={"ID":"3c993571-01f4-4d99-81a0-1c7428a5e4b3","Type":"ContainerDied","Data":"ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f"} Oct 30 13:41:13.033980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.033895 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k" event={"ID":"3c993571-01f4-4d99-81a0-1c7428a5e4b3","Type":"ContainerDied","Data":"5915de981aa94b9bbcfda783b492a0aee94233a1b625b1300e9257dae95b770e"} Oct 30 13:41:13.033980 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.033915 5515 scope.go:122] "RemoveContainer" containerID="ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f" Oct 30 13:41:13.046524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.046505 5515 scope.go:122] "RemoveContainer" containerID="ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f" Oct 30 13:41:13.046843 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:41:13.046819 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f\": container with ID starting with ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f not found: ID does not exist" containerID="ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f" Oct 30 13:41:13.046920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.046852 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f"} err="failed to get container status \"ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f\": rpc error: code = NotFound desc = could not find container \"ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f\": container with ID starting with ca3c3a337f6142735a49e26b55c28d01443cd7eb761fc9b4474fa28d4e14bd2f not found: ID does not exist" Oct 30 13:41:13.089130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.089093 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k"] Oct 30 13:41:13.102031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.101994 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d41320/test-model-d41320-kserve-7dd65f975-nq42k"] Oct 30 13:41:13.479724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:13.479693 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:14.005605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:14.005570 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c993571-01f4-4d99-81a0-1c7428a5e4b3" path="/var/lib/kubelet/pods/3c993571-01f4-4d99-81a0-1c7428a5e4b3/volumes" Oct 30 13:41:15.147090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:15.147054 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:15.349187 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:15.349156 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:17.034262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:17.034232 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:18.809339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:18.809303 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:20.487555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:20.487514 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:20.988289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:20.984085 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:22.447402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:22.447362 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:24.290777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:24.290748 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:25.986962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:25.986905 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:27.767198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:27.758048 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:28.990035 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:41:28.990000 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:41:29.588191 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:29.588160 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:31.313781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:31.313731 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:31.908439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:31.908404 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:32.112342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:32.112291 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:33.156309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:33.155564 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:34.818368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:34.818338 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:36.513886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:36.513835 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:37.586427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:37.586393 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr"] Oct 30 13:41:37.586886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:37.586599 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" containerName="istio-proxy" containerID="cri-o://d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526" gracePeriod=30 Oct 30 13:41:38.215396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:38.215358 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:39.879022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:39.878980 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:41.621665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:41.621598 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:42.887437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.887409 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:41:42.930232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930199 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-kube-api-access-76vdj\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-kube-api-access-76vdj\") pod \"2f18929d-bc55-4411-9648-cd8d0db04993\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " Oct 30 13:41:42.930232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930237 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-credential-socket\") pod \"2f18929d-bc55-4411-9648-cd8d0db04993\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " Oct 30 13:41:42.930491 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930371 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f18929d-bc55-4411-9648-cd8d0db04993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18929d-bc55-4411-9648-cd8d0db04993-istiod-ca-cert\") pod \"2f18929d-bc55-4411-9648-cd8d0db04993\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " Oct 30 13:41:42.930491 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930444 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-envoy\") pod \"2f18929d-bc55-4411-9648-cd8d0db04993\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " Oct 30 13:41:42.930491 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930469 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-istio-token\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-istio-token\") pod \"2f18929d-bc55-4411-9648-cd8d0db04993\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " Oct 30 13:41:42.930678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930510 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-certs\") pod \"2f18929d-bc55-4411-9648-cd8d0db04993\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " Oct 30 13:41:42.930678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930566 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-socket\") pod \"2f18929d-bc55-4411-9648-cd8d0db04993\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " Oct 30 13:41:42.930678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930624 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-credential-socket" pod "2f18929d-bc55-4411-9648-cd8d0db04993" (UID: "2f18929d-bc55-4411-9648-cd8d0db04993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:42.930865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930680 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-data\") pod \"2f18929d-bc55-4411-9648-cd8d0db04993\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " Oct 30 13:41:42.930865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930709 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f18929d-bc55-4411-9648-cd8d0db04993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18929d-bc55-4411-9648-cd8d0db04993-istio-podinfo\") pod \"2f18929d-bc55-4411-9648-cd8d0db04993\" (UID: \"2f18929d-bc55-4411-9648-cd8d0db04993\") " Oct 30 13:41:42.930971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930872 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-certs" pod "2f18929d-bc55-4411-9648-cd8d0db04993" (UID: "2f18929d-bc55-4411-9648-cd8d0db04993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:42.930971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.930896 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f18929d-bc55-4411-9648-cd8d0db04993-istiod-ca-cert" pod "2f18929d-bc55-4411-9648-cd8d0db04993" (UID: "2f18929d-bc55-4411-9648-cd8d0db04993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:42.931094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.931007 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-data" pod "2f18929d-bc55-4411-9648-cd8d0db04993" (UID: "2f18929d-bc55-4411-9648-cd8d0db04993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:42.931094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.931074 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-socket" pod "2f18929d-bc55-4411-9648-cd8d0db04993" (UID: "2f18929d-bc55-4411-9648-cd8d0db04993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:42.931596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.931566 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.931596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.931588 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18929d-bc55-4411-9648-cd8d0db04993-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.931766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.931602 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.931766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.931614 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.931766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.931627 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:42.932667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.932642 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-kube-api-access-76vdj" pod "2f18929d-bc55-4411-9648-cd8d0db04993" (UID: "2f18929d-bc55-4411-9648-cd8d0db04993"). InnerVolumeSpecName "kube-api-access-76vdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:42.932876 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.932861 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-istio-token" pod "2f18929d-bc55-4411-9648-cd8d0db04993" (UID: "2f18929d-bc55-4411-9648-cd8d0db04993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:42.933039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.933013 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-envoy" pod "2f18929d-bc55-4411-9648-cd8d0db04993" (UID: "2f18929d-bc55-4411-9648-cd8d0db04993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:42.933133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:42.933081 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f18929d-bc55-4411-9648-cd8d0db04993-istio-podinfo" pod "2f18929d-bc55-4411-9648-cd8d0db04993" (UID: "2f18929d-bc55-4411-9648-cd8d0db04993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:43.033535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.033498 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18929d-bc55-4411-9648-cd8d0db04993-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.033535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.033525 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.033535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.033534 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18929d-bc55-4411-9648-cd8d0db04993-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.033535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.033543 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76vdj\" (UniqueName: \"kubernetes.io/projected/2f18929d-bc55-4411-9648-cd8d0db04993-kube-api-access-76vdj\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:41:43.338842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.338805 5515 generic.go:356] "Generic (PLEG): container finished" podID="2f18929d-bc55-4411-9648-cd8d0db04993" containerID="d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526" exitCode=0 Oct 30 13:41:43.339038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.338841 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" event={"ID":"2f18929d-bc55-4411-9648-cd8d0db04993","Type":"ContainerDied","Data":"d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526"} Oct 30 13:41:43.339038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.338867 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" Oct 30 13:41:43.339038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.338910 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr" event={"ID":"2f18929d-bc55-4411-9648-cd8d0db04993","Type":"ContainerDied","Data":"8345a2444c80a267ce25abedbe9a723ff14fdcdf332ec8b2b757ec1da71fed27"} Oct 30 13:41:43.339038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.338957 5515 scope.go:122] "RemoveContainer" containerID="d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526" Oct 30 13:41:43.355504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.355483 5515 scope.go:122] "RemoveContainer" containerID="d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526" Oct 30 13:41:43.355945 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:41:43.355924 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526\": container with ID starting with d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526 not found: ID does not exist" containerID="d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526" Oct 30 13:41:43.356041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.355958 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526"} err="failed to get container status \"d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526\": rpc error: code = NotFound desc = could not find container \"d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526\": container with ID starting with d2662d16978763669097802518e78c6bba5b458423b10013cd64f30f9a8a4526 not found: ID does not exist" Oct 30 13:41:43.377607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.377581 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:43.395315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.395281 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr"] Oct 30 13:41:43.408473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:43.408447 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2d2a0b-openshift-default-79dff8cb77-ndnpr"] Oct 30 13:41:44.005066 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:44.005033 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" path="/var/lib/kubelet/pods/2f18929d-bc55-4411-9648-cd8d0db04993/volumes" Oct 30 13:41:45.147586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:45.147538 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:47.011514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:47.011483 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:48.785888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:48.785850 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:50.607612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:50.607570 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:52.291645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:52.291607 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:54.091017 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:54.090980 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:55.820837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:55.820801 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:57.959626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:57.959576 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:41:58.990564 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:41:58.990524 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:41:59.583699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:41:59.583664 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:01.210748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:01.210696 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:03.604934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:03.604889 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:04.467607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:04.467501 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:04.991986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:04.991942 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:05.218657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:05.218626 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:05.771484 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:05.771445 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:06.962806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:06.962770 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:08.726490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:08.726461 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:10.412110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:10.412062 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:12.089917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:12.089816 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:12.602126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:12.602092 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:12.819590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:12.819561 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:13.053931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:13.053900 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:14.081817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:14.081788 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:15.330217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:15.330178 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:16.215377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:16.215342 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:17.632320 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:17.632248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:19.343301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:19.343249 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:21.177066 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:21.177021 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:22.920662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:22.920625 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:24.060318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:24.060288 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc"] Oct 30 13:42:24.100561 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:24.100531 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm"] Oct 30 13:42:24.318008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:24.317918 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:24.542515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:24.542480 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:24.782290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:24.776742 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:26.437977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:26.437941 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:28.100477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:28.100432 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:28.989807 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:42:28.989773 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:42:29.061357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:29.061311 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" podUID="d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" containerName="payload-processing" containerID="cri-o://342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285" gracePeriod=25 Oct 30 13:42:29.101093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:29.101059 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" podUID="67f318bc-f058-4cd0-b681-75f77bffa1f0" containerName="payload-pre-processing" containerID="cri-o://831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f" gracePeriod=25 Oct 30 13:42:29.931738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:29.931705 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:31.614070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:31.614035 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:33.343257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:33.343211 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:35.160088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:35.160044 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:36.170130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:36.170088 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:42:36.913536 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:36.913499 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:37.932600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:37.932561 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" podUID="67f318bc-f058-4cd0-b681-75f77bffa1f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:42:37.948413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:37.948371 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" podUID="d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:42:38.782016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:38.781982 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:40.629966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:40.629919 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:42.320560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:42.320525 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:43.968752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:43.968711 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:45.722464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:45.722426 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:47.510366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:47.510321 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:47.918817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:47.918781 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" podUID="d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:42:47.918988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:47.918793 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" podUID="67f318bc-f058-4cd0-b681-75f77bffa1f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:42:49.181079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.181043 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:49.267900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.267865 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6"] Oct 30 13:42:49.268117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.268069 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" containerID="cri-o://bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664" gracePeriod=30 Oct 30 13:42:49.421023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.420997 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:42:49.439505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.439482 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-plugins-config-volume\") pod \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\" (UID: \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\") " Oct 30 13:42:49.439733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.439717 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-kube-api-access-vlr6j\" (UniqueName: \"kubernetes.io/projected/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-kube-api-access-vlr6j\") pod \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\" (UID: \"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d\") " Oct 30 13:42:49.439839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.439819 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-plugins-config-volume" pod "d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" (UID: "d1d92fb0-19e2-47d5-a770-069c5a1f2d4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:49.440437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.440415 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.441799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.441775 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-kube-api-access-vlr6j" pod "d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" (UID: "d1d92fb0-19e2-47d5-a770-069c5a1f2d4d"). InnerVolumeSpecName "kube-api-access-vlr6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:49.456324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.456302 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:42:49.541883 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.541845 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f318bc-f058-4cd0-b681-75f77bffa1f0-kube-api-access-9tncd\" (UniqueName: \"kubernetes.io/projected/67f318bc-f058-4cd0-b681-75f77bffa1f0-kube-api-access-9tncd\") pod \"67f318bc-f058-4cd0-b681-75f77bffa1f0\" (UID: \"67f318bc-f058-4cd0-b681-75f77bffa1f0\") " Oct 30 13:42:49.542136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.542115 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f318bc-f058-4cd0-b681-75f77bffa1f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f318bc-f058-4cd0-b681-75f77bffa1f0-plugins-config-volume\") pod \"67f318bc-f058-4cd0-b681-75f77bffa1f0\" (UID: \"67f318bc-f058-4cd0-b681-75f77bffa1f0\") " Oct 30 13:42:49.542624 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.542562 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f318bc-f058-4cd0-b681-75f77bffa1f0-plugins-config-volume" pod "67f318bc-f058-4cd0-b681-75f77bffa1f0" (UID: "67f318bc-f058-4cd0-b681-75f77bffa1f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:49.543029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.543005 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f318bc-f058-4cd0-b681-75f77bffa1f0-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.543112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.543038 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlr6j\" (UniqueName: \"kubernetes.io/projected/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d-kube-api-access-vlr6j\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.544251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.544220 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f318bc-f058-4cd0-b681-75f77bffa1f0-kube-api-access-9tncd" pod "67f318bc-f058-4cd0-b681-75f77bffa1f0" (UID: "67f318bc-f058-4cd0-b681-75f77bffa1f0"). InnerVolumeSpecName "kube-api-access-9tncd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:49.644858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.644830 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tncd\" (UniqueName: \"kubernetes.io/projected/67f318bc-f058-4cd0-b681-75f77bffa1f0-kube-api-access-9tncd\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:49.894733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.894696 5515 generic.go:356] "Generic (PLEG): container finished" podID="67f318bc-f058-4cd0-b681-75f77bffa1f0" containerID="831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f" exitCode=0 Oct 30 13:42:49.894733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.894733 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" Oct 30 13:42:49.894990 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.894781 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" event={"ID":"67f318bc-f058-4cd0-b681-75f77bffa1f0","Type":"ContainerDied","Data":"831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f"} Oct 30 13:42:49.894990 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.894833 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm" event={"ID":"67f318bc-f058-4cd0-b681-75f77bffa1f0","Type":"ContainerDied","Data":"669161a52ae26252a3482f27e9e8ab6ea33bcbcbc3b2833be4418d1a3d0a6775"} Oct 30 13:42:49.894990 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.894864 5515 scope.go:122] "RemoveContainer" containerID="831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f" Oct 30 13:42:49.896650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.896626 5515 generic.go:356] "Generic (PLEG): container finished" podID="d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" containerID="342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285" exitCode=0 Oct 30 13:42:49.896776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.896685 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" Oct 30 13:42:49.896776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.896704 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" event={"ID":"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d","Type":"ContainerDied","Data":"342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285"} Oct 30 13:42:49.896776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.896765 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc" event={"ID":"d1d92fb0-19e2-47d5-a770-069c5a1f2d4d","Type":"ContainerDied","Data":"cfd159ebff583f1f7db0c3cc5c0fb48ac605289f38bedd124ca381e075a856a9"} Oct 30 13:42:49.907007 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.906986 5515 scope.go:122] "RemoveContainer" containerID="831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f" Oct 30 13:42:49.907329 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:42:49.907262 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f\": container with ID starting with 831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f not found: ID does not exist" containerID="831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f" Oct 30 13:42:49.907393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.907339 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f"} err="failed to get container status \"831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f\": rpc error: code = NotFound desc = could not find container \"831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f\": container with ID starting with 831584cdb85e50711c8c8d4ff7bcc5235e6d3ccbe08377b371ffdbea6c8bea2f not found: ID does not exist" Oct 30 13:42:49.907393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.907358 5515 scope.go:122] "RemoveContainer" containerID="342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285" Oct 30 13:42:49.917503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.917487 5515 scope.go:122] "RemoveContainer" containerID="342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285" Oct 30 13:42:49.917735 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:42:49.917718 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285\": container with ID starting with 342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285 not found: ID does not exist" containerID="342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285" Oct 30 13:42:49.917787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.917740 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285"} err="failed to get container status \"342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285\": rpc error: code = NotFound desc = could not find container \"342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285\": container with ID starting with 342e44f82b7bcbeb3cc017bbe16f433e47fae0ef456ac20e106773d76486b285 not found: ID does not exist" Oct 30 13:42:49.953516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.953488 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm"] Oct 30 13:42:49.960177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:49.960152 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba646b-787f7ddf4c-vx7jm"] Oct 30 13:42:50.005406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:50.005376 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f318bc-f058-4cd0-b681-75f77bffa1f0" path="/var/lib/kubelet/pods/67f318bc-f058-4cd0-b681-75f77bffa1f0/volumes" Oct 30 13:42:50.005753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:50.005740 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc"] Oct 30 13:42:50.005804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:50.005766 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba646b-597b979dd4-4fxpc"] Oct 30 13:42:50.898288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:50.897917 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:52.003465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:52.003426 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" path="/var/lib/kubelet/pods/d1d92fb0-19e2-47d5-a770-069c5a1f2d4d/volumes" Oct 30 13:42:52.629949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:52.629917 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:54.468004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.467968 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:54.555053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.554992 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:42:54.618209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618179 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istiod-ca-cert\") pod \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " Oct 30 13:42:54.618438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618234 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-socket\") pod \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " Oct 30 13:42:54.618438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618305 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-data\") pod \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " Oct 30 13:42:54.618438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618357 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-kube-api-access-xshpz\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-kube-api-access-xshpz\") pod \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " Oct 30 13:42:54.618438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618408 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-token\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-token\") pod \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " Oct 30 13:42:54.618674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618510 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-podinfo\") pod \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " Oct 30 13:42:54.618674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618561 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-certs\") pod \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " Oct 30 13:42:54.618674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618643 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-credential-socket\") pod \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " Oct 30 13:42:54.618835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618719 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-envoy\") pod \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\" (UID: \"47806efb-6a33-4d7b-8b2f-a0eb42dcb169\") " Oct 30 13:42:54.618979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618556 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-socket" pod "47806efb-6a33-4d7b-8b2f-a0eb42dcb169" (UID: "47806efb-6a33-4d7b-8b2f-a0eb42dcb169"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:54.618979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618619 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istiod-ca-cert" pod "47806efb-6a33-4d7b-8b2f-a0eb42dcb169" (UID: "47806efb-6a33-4d7b-8b2f-a0eb42dcb169"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:54.618979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.618860 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-data" pod "47806efb-6a33-4d7b-8b2f-a0eb42dcb169" (UID: "47806efb-6a33-4d7b-8b2f-a0eb42dcb169"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:54.619176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.619036 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-certs" pod "47806efb-6a33-4d7b-8b2f-a0eb42dcb169" (UID: "47806efb-6a33-4d7b-8b2f-a0eb42dcb169"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:54.619575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.619546 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-credential-socket" pod "47806efb-6a33-4d7b-8b2f-a0eb42dcb169" (UID: "47806efb-6a33-4d7b-8b2f-a0eb42dcb169"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:54.619689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.619629 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.619689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.619651 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.619689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.619664 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.619689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.619677 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.621478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.621448 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-kube-api-access-xshpz" pod "47806efb-6a33-4d7b-8b2f-a0eb42dcb169" (UID: "47806efb-6a33-4d7b-8b2f-a0eb42dcb169"). InnerVolumeSpecName "kube-api-access-xshpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:54.622200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.622135 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-token" pod "47806efb-6a33-4d7b-8b2f-a0eb42dcb169" (UID: "47806efb-6a33-4d7b-8b2f-a0eb42dcb169"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:54.622200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.622173 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-envoy" pod "47806efb-6a33-4d7b-8b2f-a0eb42dcb169" (UID: "47806efb-6a33-4d7b-8b2f-a0eb42dcb169"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:54.622379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.622221 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-podinfo" pod "47806efb-6a33-4d7b-8b2f-a0eb42dcb169" (UID: "47806efb-6a33-4d7b-8b2f-a0eb42dcb169"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:42:54.721630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.721600 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xshpz\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-kube-api-access-xshpz\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.721630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.721627 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.721630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.721636 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.721864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.721645 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.721864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.721654 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47806efb-6a33-4d7b-8b2f-a0eb42dcb169-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.945656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.945620 5515 generic.go:356] "Generic (PLEG): container finished" podID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerID="bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664" exitCode=0 Oct 30 13:42:54.945878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.945675 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" Oct 30 13:42:54.945878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.945703 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" event={"ID":"47806efb-6a33-4d7b-8b2f-a0eb42dcb169","Type":"ContainerDied","Data":"bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664"} Oct 30 13:42:54.945878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.945767 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6" event={"ID":"47806efb-6a33-4d7b-8b2f-a0eb42dcb169","Type":"ContainerDied","Data":"3332a55796e7b9d91b85ff7904f43617ec75c91eb8f5042bb08e2b18a28c3255"} Oct 30 13:42:54.945878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.945793 5515 scope.go:122] "RemoveContainer" containerID="bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664" Oct 30 13:42:54.958101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.958079 5515 scope.go:122] "RemoveContainer" containerID="bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664" Oct 30 13:42:54.958407 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:42:54.958386 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664\": container with ID starting with bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664 not found: ID does not exist" containerID="bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664" Oct 30 13:42:54.958489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:54.958419 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664"} err="failed to get container status \"bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664\": rpc error: code = NotFound desc = could not find container \"bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664\": container with ID starting with bca0559add114859d81b07e7c1827a7a66d637cc1b3798b9403633a7052c6664 not found: ID does not exist" Oct 30 13:42:55.001292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:55.001250 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6"] Oct 30 13:42:55.013813 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:55.013788 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-6jgv6"] Oct 30 13:42:56.003891 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:56.003859 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" path="/var/lib/kubelet/pods/47806efb-6a33-4d7b-8b2f-a0eb42dcb169/volumes" Oct 30 13:42:56.171865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:56.171826 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:58.189926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:58.189889 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:58.991657 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:42:58.991620 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:42:59.646822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.646786 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b"] Oct 30 13:42:59.647971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.647940 5515 cpu_manager.go:414] "containerMap: removing container" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" containerName="istio-proxy" Oct 30 13:42:59.647971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.647968 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" containerName="istio-proxy" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.647996 5515 cpu_manager.go:414] "containerMap: removing container" podUID="67f318bc-f058-4cd0-b681-75f77bffa1f0" containerName="payload-pre-processing" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648004 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f318bc-f058-4cd0-b681-75f77bffa1f0" containerName="payload-pre-processing" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648043 5515 cpu_manager.go:414] "containerMap: removing container" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648048 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648060 5515 cpu_manager.go:414] "containerMap: removing container" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648065 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648081 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" containerName="payload-processing" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648086 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" containerName="payload-processing" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648103 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3c993571-01f4-4d99-81a0-1c7428a5e4b3" containerName="main" Oct 30 13:42:59.648168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648108 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c993571-01f4-4d99-81a0-1c7428a5e4b3" containerName="main" Oct 30 13:42:59.648504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648258 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="47806efb-6a33-4d7b-8b2f-a0eb42dcb169" containerName="istio-proxy" Oct 30 13:42:59.648504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648283 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="42262685-7312-4ac0-8479-be793dab101c" containerName="istio-proxy" Oct 30 13:42:59.648504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648291 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1d92fb0-19e2-47d5-a770-069c5a1f2d4d" containerName="payload-processing" Oct 30 13:42:59.648504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648304 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c993571-01f4-4d99-81a0-1c7428a5e4b3" containerName="main" Oct 30 13:42:59.648504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648313 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f18929d-bc55-4411-9648-cd8d0db04993" containerName="istio-proxy" Oct 30 13:42:59.648504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.648323 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f318bc-f058-4cd0-b681-75f77bffa1f0" containerName="payload-pre-processing" Oct 30 13:42:59.653749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.653727 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.676002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.675965 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-025c7f-openshift-default-dockercfg-ztrnc\"" Oct 30 13:42:59.693769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.693724 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-certs\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.693948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.693774 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-token\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.693948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.693939 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-envoy\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.694058 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.693782 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b"] Oct 30 13:42:59.694140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.693990 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-credential-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.694259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.694239 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istiod-ca-cert\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.694347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.694321 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-data\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.694403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.694351 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.694459 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.694412 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-podinfo\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.694506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.694457 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xctw\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-kube-api-access-9xctw\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.795588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.795553 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-envoy\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.795761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.795607 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-credential-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.795761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.795672 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istiod-ca-cert\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.795761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.795722 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-data\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.795761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.795752 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.795988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.795807 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-podinfo\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.795988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.795852 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xctw\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-kube-api-access-9xctw\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.795988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.795917 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-certs\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.795988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.795947 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-token\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.796260 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.796233 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-credential-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.796488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.796465 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-data\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.796647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.796623 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-certs\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.796695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.796625 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istiod-ca-cert\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.796888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.796868 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-socket\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.798059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.798038 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-envoy\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.798365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.798345 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-podinfo\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.821282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.821251 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xctw\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-kube-api-access-9xctw\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.823345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.823327 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-token\") pod \"e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:42:59.920933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.920780 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:42:59.970155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:42:59.970110 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:43:00.303512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:00.303485 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b"] Oct 30 13:43:00.303642 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:43:00.303607 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a6b23d_d1b5_45f1_bbca_3185a13f5bf3.slice/crio-ef75176488c74bb55e6103d93bb865c87f8643169711386a1d4bb234b47bc680 WatchSource:0}: Error finding container ef75176488c74bb55e6103d93bb865c87f8643169711386a1d4bb234b47bc680: Status 404 returned error can't find the container with id ef75176488c74bb55e6103d93bb865c87f8643169711386a1d4bb234b47bc680 Oct 30 13:43:00.306781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:00.306748 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:43:00.306873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:00.306814 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:43:00.306873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:00.306845 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:43:01.017290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:01.016247 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" event={"ID":"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3","Type":"ContainerStarted","Data":"d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28"} Oct 30 13:43:01.017290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:01.016328 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" event={"ID":"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3","Type":"ContainerStarted","Data":"ef75176488c74bb55e6103d93bb865c87f8643169711386a1d4bb234b47bc680"} Oct 30 13:43:01.079792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:01.079735 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" podStartSLOduration=2.079721584 podStartE2EDuration="2.079721584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:01.077177988 +0000 UTC m=+4199.203978887" watchObservedRunningTime="2026-10-30 13:43:01.079721584 +0000 UTC m=+4199.206522482" Oct 30 13:43:01.723803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:01.723770 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:01.970902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:01.970864 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:43:01.972445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:01.972420 5515 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 13:43:01.972577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:01.972465 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 13:43:02.971289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:02.971231 5515 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 13:43:02.971717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:02.971314 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 13:43:03.541012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:03.540978 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:03.970633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:03.970598 5515 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 13:43:03.970803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:03.970677 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 13:43:04.970557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:04.970522 5515 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 13:43:04.970954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:04.970589 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 13:43:05.260927 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:05.260829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:05.971212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:05.971173 5515 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 13:43:05.971694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:05.971231 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 13:43:06.989630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:06.989598 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:43:06.990123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:06.990082 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:43:06.991206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:06.991084 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:43:07.003901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:07.003862 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:08.712836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:08.712796 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:10.414736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:10.414701 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:12.225856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:12.225826 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:13.984764 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:13.984727 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:15.308898 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.308864 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv"] Oct 30 13:43:15.309420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.309158 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" podUID="ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" containerName="maas-api" containerID="cri-o://0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5" gracePeriod=30 Oct 30 13:43:15.384360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.384307 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" podUID="ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.246:8443/health\": dial tcp 10.134.0.246:8443: connect: connection refused" Oct 30 13:43:15.469726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.469682 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:15.506666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.506626 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn"] Oct 30 13:43:15.507088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.506971 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" containerID="cri-o://f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1" gracePeriod=30 Oct 30 13:43:15.610987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.610959 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:43:15.693935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.693283 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-openshift-service-ca\") pod \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " Oct 30 13:43:15.693935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.693402 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-kube-api-access-rhfjk\" (UniqueName: \"kubernetes.io/projected/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-kube-api-access-rhfjk\") pod \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " Oct 30 13:43:15.693935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.693625 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-maas-api-tls\") pod \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " Oct 30 13:43:15.693935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.693682 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-metrics-certs\") pod \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\" (UID: \"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6\") " Oct 30 13:43:15.693935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.693809 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-openshift-service-ca" pod "ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" (UID: "ba10fcf2-1c37-46c7-ba6c-b3c0817407d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:15.695389 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.695035 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:15.697228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.697067 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-kube-api-access-rhfjk" pod "ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" (UID: "ba10fcf2-1c37-46c7-ba6c-b3c0817407d6"). InnerVolumeSpecName "kube-api-access-rhfjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:15.697965 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.697936 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:15.703247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.703205 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-metrics-certs" pod "ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" (UID: "ba10fcf2-1c37-46c7-ba6c-b3c0817407d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:15.704132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.704096 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-maas-api-tls" pod "ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" (UID: "ba10fcf2-1c37-46c7-ba6c-b3c0817407d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:43:15.797757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.797713 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhfjk\" (UniqueName: \"kubernetes.io/projected/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-kube-api-access-rhfjk\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:15.797757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.797757 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:15.798079 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.797773 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:15.971030 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:15.970985 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:16.161483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.161446 5515 generic.go:356] "Generic (PLEG): container finished" podID="ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" containerID="0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5" exitCode=0 Oct 30 13:43:16.161653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.161500 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" Oct 30 13:43:16.161653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.161606 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" event={"ID":"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6","Type":"ContainerDied","Data":"0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5"} Oct 30 13:43:16.161770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.161653 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv" event={"ID":"ba10fcf2-1c37-46c7-ba6c-b3c0817407d6","Type":"ContainerDied","Data":"3dd223d26397d52fea2bf751b942834d47491c4d60a71e0390ae327998e5852f"} Oct 30 13:43:16.161770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.161684 5515 scope.go:122] "RemoveContainer" containerID="0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5" Oct 30 13:43:16.175238 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.175217 5515 scope.go:122] "RemoveContainer" containerID="0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5" Oct 30 13:43:16.175626 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:43:16.175598 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5\": container with ID starting with 0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5 not found: ID does not exist" containerID="0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5" Oct 30 13:43:16.175770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.175631 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5"} err="failed to get container status \"0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5\": rpc error: code = NotFound desc = could not find container \"0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5\": container with ID starting with 0aa76c25c95f947bbd993b68ba08a28a19404742a7923b7fdb991902253beed5 not found: ID does not exist" Oct 30 13:43:16.194815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.194783 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:16.218799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.218765 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv"] Oct 30 13:43:16.231236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.231167 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ba646b-c88d55677-gfqsv"] Oct 30 13:43:16.683316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:16.683254 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:18.004722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:18.004686 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" path="/var/lib/kubelet/pods/ba10fcf2-1c37-46c7-ba6c-b3c0817407d6/volumes" Oct 30 13:43:18.066527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:18.066488 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:19.748461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:19.748428 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:20.805617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.805593 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:43:20.892233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.892199 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-podinfo\") pod \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " Oct 30 13:43:20.892590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.892564 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-credential-socket\") pod \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " Oct 30 13:43:20.892787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.892769 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istiod-ca-cert\") pod \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " Oct 30 13:43:20.892975 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.892949 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-token\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-token\") pod \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " Oct 30 13:43:20.893104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.892990 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-certs\") pod \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " Oct 30 13:43:20.893104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.893020 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-socket\") pod \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " Oct 30 13:43:20.893104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.893075 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-envoy\") pod \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " Oct 30 13:43:20.893336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.893110 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-kube-api-access-lwchv\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-kube-api-access-lwchv\") pod \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " Oct 30 13:43:20.893422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.893401 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-data\") pod \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\" (UID: \"aa52c12b-284b-4a79-90a6-9ed858ed0fb8\") " Oct 30 13:43:20.893937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.893910 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-data" pod "aa52c12b-284b-4a79-90a6-9ed858ed0fb8" (UID: "aa52c12b-284b-4a79-90a6-9ed858ed0fb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:20.894570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.894548 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-certs" pod "aa52c12b-284b-4a79-90a6-9ed858ed0fb8" (UID: "aa52c12b-284b-4a79-90a6-9ed858ed0fb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:20.894869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.894842 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:20.894960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.894875 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:20.895064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.895040 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-credential-socket" pod "aa52c12b-284b-4a79-90a6-9ed858ed0fb8" (UID: "aa52c12b-284b-4a79-90a6-9ed858ed0fb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:20.895486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.895465 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istiod-ca-cert" pod "aa52c12b-284b-4a79-90a6-9ed858ed0fb8" (UID: "aa52c12b-284b-4a79-90a6-9ed858ed0fb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:20.896732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.896699 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-podinfo" pod "aa52c12b-284b-4a79-90a6-9ed858ed0fb8" (UID: "aa52c12b-284b-4a79-90a6-9ed858ed0fb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:43:20.897462 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.897434 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-envoy" pod "aa52c12b-284b-4a79-90a6-9ed858ed0fb8" (UID: "aa52c12b-284b-4a79-90a6-9ed858ed0fb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:20.897671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.897649 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-socket" pod "aa52c12b-284b-4a79-90a6-9ed858ed0fb8" (UID: "aa52c12b-284b-4a79-90a6-9ed858ed0fb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:20.899367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.899342 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-kube-api-access-lwchv" pod "aa52c12b-284b-4a79-90a6-9ed858ed0fb8" (UID: "aa52c12b-284b-4a79-90a6-9ed858ed0fb8"). InnerVolumeSpecName "kube-api-access-lwchv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:20.899522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.899501 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-token" pod "aa52c12b-284b-4a79-90a6-9ed858ed0fb8" (UID: "aa52c12b-284b-4a79-90a6-9ed858ed0fb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:20.996153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.996119 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:20.996153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.996147 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:20.996153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.996156 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:20.996430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.996165 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:20.996430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.996174 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:20.996430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.996183 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:20.996430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:20.996192 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwchv\" (UniqueName: \"kubernetes.io/projected/aa52c12b-284b-4a79-90a6-9ed858ed0fb8-kube-api-access-lwchv\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:21.203282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.203242 5515 generic.go:356] "Generic (PLEG): container finished" podID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerID="f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1" exitCode=0 Oct 30 13:43:21.203466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.203314 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" Oct 30 13:43:21.203466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.203331 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" event={"ID":"aa52c12b-284b-4a79-90a6-9ed858ed0fb8","Type":"ContainerDied","Data":"f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1"} Oct 30 13:43:21.203466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.203384 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn" event={"ID":"aa52c12b-284b-4a79-90a6-9ed858ed0fb8","Type":"ContainerDied","Data":"fb3bfbd4880a39ea6de24c3fb8476306351c524de33fd3d15c0a4425282a874f"} Oct 30 13:43:21.203466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.203408 5515 scope.go:122] "RemoveContainer" containerID="f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1" Oct 30 13:43:21.215893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.215876 5515 scope.go:122] "RemoveContainer" containerID="f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1" Oct 30 13:43:21.216151 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:43:21.216134 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1\": container with ID starting with f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1 not found: ID does not exist" containerID="f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1" Oct 30 13:43:21.216193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.216158 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1"} err="failed to get container status \"f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1\": rpc error: code = NotFound desc = could not find container \"f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1\": container with ID starting with f2c642589557d8233ddc41695a20bcd5a6f6cf18f182e3b358640e678f30bcd1 not found: ID does not exist" Oct 30 13:43:21.258143 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.258116 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn"] Oct 30 13:43:21.262631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.262613 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p95xn"] Oct 30 13:43:21.495596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:21.495499 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:22.015420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:22.015385 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" path="/var/lib/kubelet/pods/aa52c12b-284b-4a79-90a6-9ed858ed0fb8/volumes" Oct 30 13:43:23.284897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:23.284853 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:23.523788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:23.523730 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:23.777068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:23.777035 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:23.779794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:23.779770 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll"] Oct 30 13:43:23.779989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:23.779969 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" containerID="cri-o://426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d" gracePeriod=30 Oct 30 13:43:23.996309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:23.991060 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:24.824108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:24.824061 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:25.090638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:25.090561 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:25.315114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:25.315083 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:26.915867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:26.915836 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:28.705112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:28.705079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:29.000728 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:43:29.000677 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:43:29.076089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.076052 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:43:29.129597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.129563 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-socket\") pod \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " Oct 30 13:43:29.129759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.129709 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-data\") pod \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " Oct 30 13:43:29.129807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.129788 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-certs\") pod \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " Oct 30 13:43:29.129860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.129839 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-kube-api-access-kqlzt\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-kube-api-access-kqlzt\") pod \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " Oct 30 13:43:29.129902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.129860 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-socket" pod "abf0cda3-bd5d-4ab1-9695-2e435758cc6e" (UID: "abf0cda3-bd5d-4ab1-9695-2e435758cc6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:29.129902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.129885 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-envoy\") pod \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " Oct 30 13:43:29.130001 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.129985 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-podinfo\") pod \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " Oct 30 13:43:29.130059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.129985 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-data" pod "abf0cda3-bd5d-4ab1-9695-2e435758cc6e" (UID: "abf0cda3-bd5d-4ab1-9695-2e435758cc6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:29.130059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130027 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-token\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-token\") pod \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " Oct 30 13:43:29.130059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130049 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-certs" pod "abf0cda3-bd5d-4ab1-9695-2e435758cc6e" (UID: "abf0cda3-bd5d-4ab1-9695-2e435758cc6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:29.130225 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130063 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-credential-socket\") pod \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " Oct 30 13:43:29.130225 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130130 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istiod-ca-cert\") pod \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\" (UID: \"abf0cda3-bd5d-4ab1-9695-2e435758cc6e\") " Oct 30 13:43:29.130396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130262 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-credential-socket" pod "abf0cda3-bd5d-4ab1-9695-2e435758cc6e" (UID: "abf0cda3-bd5d-4ab1-9695-2e435758cc6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:29.130542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130519 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istiod-ca-cert" pod "abf0cda3-bd5d-4ab1-9695-2e435758cc6e" (UID: "abf0cda3-bd5d-4ab1-9695-2e435758cc6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:29.130966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130940 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:29.131063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130972 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:29.131063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130986 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:29.131063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.130998 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:29.131063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.131016 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:29.132218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.132193 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-token" pod "abf0cda3-bd5d-4ab1-9695-2e435758cc6e" (UID: "abf0cda3-bd5d-4ab1-9695-2e435758cc6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:29.132218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.132207 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-podinfo" pod "abf0cda3-bd5d-4ab1-9695-2e435758cc6e" (UID: "abf0cda3-bd5d-4ab1-9695-2e435758cc6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:43:29.132380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.132242 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-envoy" pod "abf0cda3-bd5d-4ab1-9695-2e435758cc6e" (UID: "abf0cda3-bd5d-4ab1-9695-2e435758cc6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:29.132380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.132328 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-kube-api-access-kqlzt" pod "abf0cda3-bd5d-4ab1-9695-2e435758cc6e" (UID: "abf0cda3-bd5d-4ab1-9695-2e435758cc6e"). InnerVolumeSpecName "kube-api-access-kqlzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:29.232451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.232375 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqlzt\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-kube-api-access-kqlzt\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:29.232451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.232403 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:29.232451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.232412 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:29.232451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.232419 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf0cda3-bd5d-4ab1-9695-2e435758cc6e-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:43:29.269846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.269814 5515 generic.go:356] "Generic (PLEG): container finished" podID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerID="426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d" exitCode=0 Oct 30 13:43:29.270151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.269865 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" Oct 30 13:43:29.270151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.269889 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" event={"ID":"abf0cda3-bd5d-4ab1-9695-2e435758cc6e","Type":"ContainerDied","Data":"426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d"} Oct 30 13:43:29.270151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.269944 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll" event={"ID":"abf0cda3-bd5d-4ab1-9695-2e435758cc6e","Type":"ContainerDied","Data":"0f4b3f8bf6648d5ec1ce27b1126e828166c83ad876a0283e2150aee106ce0728"} Oct 30 13:43:29.270151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.269965 5515 scope.go:122] "RemoveContainer" containerID="426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d" Oct 30 13:43:29.281668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.281650 5515 scope.go:122] "RemoveContainer" containerID="426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d" Oct 30 13:43:29.281950 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:43:29.281932 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d\": container with ID starting with 426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d not found: ID does not exist" containerID="426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d" Oct 30 13:43:29.281994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.281958 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d"} err="failed to get container status \"426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d\": rpc error: code = NotFound desc = could not find container \"426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d\": container with ID starting with 426f2738c82bfdb6c960b12861054f4b78be1ea60170c0322ed6d5c5066e925d not found: ID does not exist" Oct 30 13:43:29.324953 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.324917 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll"] Oct 30 13:43:29.331175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:29.331150 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ba646b-openshift-default-5b5998b9ff-4hfll"] Oct 30 13:43:30.003723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:30.003692 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" path="/var/lib/kubelet/pods/abf0cda3-bd5d-4ab1-9695-2e435758cc6e/volumes" Oct 30 13:43:30.423867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:30.423818 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:32.097734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:32.097700 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:33.922170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:33.922131 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:35.611606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:35.611570 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:37.229341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:37.229252 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:38.984544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:38.984258 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:40.650714 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:40.650680 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:42.274913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:42.274819 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:44.024592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:44.024559 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:45.903689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:45.903655 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:47.539743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:47.539712 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:49.406839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:49.406794 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:51.039827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:51.039787 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:52.647464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:52.647429 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:53.886962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:53.886934 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:43:54.448555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:54.448505 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:56.208506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:56.208463 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:57.924449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:57.924417 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:43:58.996109 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:43:58.996070 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:43:59.739369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:43:59.739335 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:01.526965 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:01.526932 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:01.769157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:01.769122 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:03.285734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:03.285697 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:05.030117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:05.030086 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:06.657978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:06.657942 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:08.391454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:08.391405 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:10.124358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:10.124326 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:11.854340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:11.854310 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:13.531890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:13.531861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:15.102951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:15.102916 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:15.575402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:15.575373 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:16.728249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:16.728217 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:18.470774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:18.470743 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:20.136917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:20.136888 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:21.811001 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:21.810963 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:22.052709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:22.052665 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:22.333085 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:22.330699 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:23.772672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:23.772640 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:25.605233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:25.605203 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:27.319906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:27.319841 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:29.026682 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:44:29.026645 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:44:29.029807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:29.029765 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:30.793722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.793626 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph"] Oct 30 13:44:30.795415 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.795391 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" containerName="maas-api" Oct 30 13:44:30.795673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.795654 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" containerName="maas-api" Oct 30 13:44:30.796016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.795966 5515 cpu_manager.go:414] "containerMap: removing container" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" Oct 30 13:44:30.796233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.796203 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" Oct 30 13:44:30.796950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.796921 5515 cpu_manager.go:414] "containerMap: removing container" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" Oct 30 13:44:30.796950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.796949 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" Oct 30 13:44:30.797210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.797184 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa52c12b-284b-4a79-90a6-9ed858ed0fb8" containerName="istio-proxy" Oct 30 13:44:30.797342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.797217 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf0cda3-bd5d-4ab1-9695-2e435758cc6e" containerName="istio-proxy" Oct 30 13:44:30.797342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.797227 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba10fcf2-1c37-46c7-ba6c-b3c0817407d6" containerName="maas-api" Oct 30 13:44:30.802600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.802477 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp"] Oct 30 13:44:30.803604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.802488 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.820289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.817213 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.822771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.820611 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph"] Oct 30 13:44:30.824692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.824662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:30.825874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.825850 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp"] Oct 30 13:44:30.889395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889360 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lf5\" (UniqueName: \"kubernetes.io/projected/2876231a-a44b-4a8f-b462-4d30201f90d5-kube-api-access-k4lf5\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.889395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889399 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.889662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889438 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.889662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889480 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2876231a-a44b-4a8f-b462-4d30201f90d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.889662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889642 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.889831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889693 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.889892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889828 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.889892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889876 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.890003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889913 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.890003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.889996 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.890109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.890031 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.890109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.890059 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.890218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.890132 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.890289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.890225 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.890356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.890286 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mklgt\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-kube-api-access-mklgt\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.890438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.890359 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.890438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.890407 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.890438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.890434 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992047 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992096 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992113 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992188 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lf5\" (UniqueName: \"kubernetes.io/projected/2876231a-a44b-4a8f-b462-4d30201f90d5-kube-api-access-k4lf5\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.992367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992224 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992287 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.992367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992310 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2876231a-a44b-4a8f-b462-4d30201f90d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.992367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992348 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992391 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992487 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992532 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992561 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992576 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.992717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992624 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992668 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.992717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992705 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.992717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992712 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.993244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992751 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.993244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992787 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.993244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992819 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.993244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992862 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.993244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992877 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.993244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992933 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mklgt\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-kube-api-access-mklgt\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.993244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.992955 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.993244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.993102 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.993244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.993223 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.993666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.993326 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2876231a-a44b-4a8f-b462-4d30201f90d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.993666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.993405 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.995123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.995100 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:30.995328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.995311 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.995588 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.995568 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:30.995773 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:30.995755 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:31.015962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.015935 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:31.016173 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.016152 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lf5\" (UniqueName: \"kubernetes.io/projected/2876231a-a44b-4a8f-b462-4d30201f90d5-kube-api-access-k4lf5\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:31.016521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.016503 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2876231a-a44b-4a8f-b462-4d30201f90d5-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ljfbp\" (UID: \"2876231a-a44b-4a8f-b462-4d30201f90d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:31.028720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.028693 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mklgt\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-kube-api-access-mklgt\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vkjph\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:31.138625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.138588 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:31.152286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.152249 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:31.461224 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:44:31.461191 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded15e5d5_20e0_4eb1_8fd0_4905858cbaea.slice/crio-69dbeb65c6754d33da2535eb93922324d63387d65149ee4c6e64408636b3963e WatchSource:0}: Error finding container 69dbeb65c6754d33da2535eb93922324d63387d65149ee4c6e64408636b3963e: Status 404 returned error can't find the container with id 69dbeb65c6754d33da2535eb93922324d63387d65149ee4c6e64408636b3963e Oct 30 13:44:31.462143 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.462025 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph"] Oct 30 13:44:31.465454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.465411 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:31.465549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.465500 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:31.465590 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.465547 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:31.480469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.480441 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp"] Oct 30 13:44:31.482930 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:44:31.482904 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2876231a_a44b_4a8f_b462_4d30201f90d5.slice/crio-3ed239c4010139a6c3aae0e54d34d76895697a739eecf138cef4636e58c20381 WatchSource:0}: Error finding container 3ed239c4010139a6c3aae0e54d34d76895697a739eecf138cef4636e58c20381: Status 404 returned error can't find the container with id 3ed239c4010139a6c3aae0e54d34d76895697a739eecf138cef4636e58c20381 Oct 30 13:44:31.487436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.487402 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:31.487550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.487470 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:31.487550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.487501 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:44:31.900974 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.900934 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" event={"ID":"2876231a-a44b-4a8f-b462-4d30201f90d5","Type":"ContainerStarted","Data":"84356a11cfbe0c4acaeb8993316b40de5a2a15092a7a07de34f501b9a743af2c"} Oct 30 13:44:31.901530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.901005 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" event={"ID":"2876231a-a44b-4a8f-b462-4d30201f90d5","Type":"ContainerStarted","Data":"3ed239c4010139a6c3aae0e54d34d76895697a739eecf138cef4636e58c20381"} Oct 30 13:44:31.903645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.903554 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" event={"ID":"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea","Type":"ContainerStarted","Data":"3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74"} Oct 30 13:44:31.903645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.903604 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" event={"ID":"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea","Type":"ContainerStarted","Data":"69dbeb65c6754d33da2535eb93922324d63387d65149ee4c6e64408636b3963e"} Oct 30 13:44:31.961873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:31.961814 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" podStartSLOduration=1.9617970439999999 podStartE2EDuration="1.961797044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:31.959681442 +0000 UTC m=+4290.086482340" watchObservedRunningTime="2026-10-30 13:44:31.961797044 +0000 UTC m=+4290.088597936" Oct 30 13:44:32.029045 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.028988 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podStartSLOduration=2.028974464 podStartE2EDuration="2.028974464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:32.017234563 +0000 UTC m=+4290.144035458" watchObservedRunningTime="2026-10-30 13:44:32.028974464 +0000 UTC m=+4290.155775361" Oct 30 13:44:32.139822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.139787 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:32.141963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.141928 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vkjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 13:44:32.142128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.141994 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 13:44:32.153075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.153010 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:32.154799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.154777 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ljfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 13:44:32.154875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.154822 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" podUID="2876231a-a44b-4a8f-b462-4d30201f90d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 13:44:32.589629 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.589523 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:32.956857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.956821 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh"] Oct 30 13:44:32.974762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:32.974727 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm"] Oct 30 13:44:33.164541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:33.162878 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vkjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 13:44:33.164541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:33.162936 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 13:44:33.164541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:33.164184 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ljfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 13:44:33.164541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:33.164229 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" podUID="2876231a-a44b-4a8f-b462-4d30201f90d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 13:44:33.245470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:33.245384 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:33.458658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:33.458621 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:34.141295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:34.140625 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vkjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 13:44:34.141295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:34.140684 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 13:44:34.152625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:34.152593 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ljfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 13:44:34.152784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:34.152639 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" podUID="2876231a-a44b-4a8f-b462-4d30201f90d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 13:44:34.382594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:34.382557 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:35.145112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:35.144728 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vkjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 13:44:35.145112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:35.144790 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 13:44:35.152698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:35.152669 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ljfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 13:44:35.152861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:35.152735 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" podUID="2876231a-a44b-4a8f-b462-4d30201f90d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 13:44:36.090400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:36.090375 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:36.143318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:36.143249 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vkjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 13:44:36.143644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:36.143595 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 13:44:36.152733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:36.152702 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ljfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 13:44:36.153102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:36.152755 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" podUID="2876231a-a44b-4a8f-b462-4d30201f90d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 13:44:37.141757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:37.141718 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vkjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 13:44:37.142066 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:37.142033 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 13:44:37.153599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:37.153574 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ljfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 13:44:37.153998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:37.153975 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" podUID="2876231a-a44b-4a8f-b462-4d30201f90d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 13:44:37.967359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:37.958449 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" podUID="380c7464-dcec-4c9a-9bce-d54738f6073e" containerName="payload-processing" containerID="cri-o://7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8" gracePeriod=25 Oct 30 13:44:37.979213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:37.975816 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" podUID="6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" containerName="payload-pre-processing" containerID="cri-o://244cfd869ada591cb80cfb5a60affd32a4d67c95fd7b567f69c37c7ab06372cc" gracePeriod=25 Oct 30 13:44:38.015586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.015548 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:38.142107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.141646 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vkjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 13:44:38.142107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.141706 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 13:44:38.153158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.153121 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ljfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 13:44:38.153361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.153184 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" podUID="2876231a-a44b-4a8f-b462-4d30201f90d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 13:44:38.360806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.360423 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:44:38.467456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.467421 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380c7464-dcec-4c9a-9bce-d54738f6073e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380c7464-dcec-4c9a-9bce-d54738f6073e-plugins-config-volume\") pod \"380c7464-dcec-4c9a-9bce-d54738f6073e\" (UID: \"380c7464-dcec-4c9a-9bce-d54738f6073e\") " Oct 30 13:44:38.467936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.467902 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380c7464-dcec-4c9a-9bce-d54738f6073e-plugins-config-volume" pod "380c7464-dcec-4c9a-9bce-d54738f6073e" (UID: "380c7464-dcec-4c9a-9bce-d54738f6073e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:38.468224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.468202 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380c7464-dcec-4c9a-9bce-d54738f6073e-kube-api-access-68mg8\" (UniqueName: \"kubernetes.io/projected/380c7464-dcec-4c9a-9bce-d54738f6073e-kube-api-access-68mg8\") pod \"380c7464-dcec-4c9a-9bce-d54738f6073e\" (UID: \"380c7464-dcec-4c9a-9bce-d54738f6073e\") " Oct 30 13:44:38.469601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.469577 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380c7464-dcec-4c9a-9bce-d54738f6073e-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:44:38.476864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.476828 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380c7464-dcec-4c9a-9bce-d54738f6073e-kube-api-access-68mg8" pod "380c7464-dcec-4c9a-9bce-d54738f6073e" (UID: "380c7464-dcec-4c9a-9bce-d54738f6073e"). InnerVolumeSpecName "kube-api-access-68mg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:38.571502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:38.571421 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68mg8\" (UniqueName: \"kubernetes.io/projected/380c7464-dcec-4c9a-9bce-d54738f6073e-kube-api-access-68mg8\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:44:39.086383 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.086340 5515 generic.go:356] "Generic (PLEG): container finished" podID="380c7464-dcec-4c9a-9bce-d54738f6073e" containerID="7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8" exitCode=0 Oct 30 13:44:39.086583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.086401 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" Oct 30 13:44:39.086583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.086415 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" event={"ID":"380c7464-dcec-4c9a-9bce-d54738f6073e","Type":"ContainerDied","Data":"7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8"} Oct 30 13:44:39.086583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.086467 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh" event={"ID":"380c7464-dcec-4c9a-9bce-d54738f6073e","Type":"ContainerDied","Data":"5e39303eb0589a09bd5e273766d9a6bcdfda92a6a2deafa449b0cf26fe5ebf9b"} Oct 30 13:44:39.086583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.086500 5515 scope.go:122] "RemoveContainer" containerID="7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8" Oct 30 13:44:39.106481 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.106453 5515 scope.go:122] "RemoveContainer" containerID="7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8" Oct 30 13:44:39.106823 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:44:39.106800 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8\": container with ID starting with 7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8 not found: ID does not exist" containerID="7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8" Oct 30 13:44:39.106936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.106831 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8"} err="failed to get container status \"7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8\": rpc error: code = NotFound desc = could not find container \"7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8\": container with ID starting with 7b544db08d398eb7ce3f1da6f9fcfe601b18e853c8798ae1969017bcc21700b8 not found: ID does not exist" Oct 30 13:44:39.141521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.141482 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vkjph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 13:44:39.141722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.141554 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 13:44:39.150036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.150002 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh"] Oct 30 13:44:39.152659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.152628 5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ljfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 13:44:39.152856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.152830 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" podUID="2876231a-a44b-4a8f-b462-4d30201f90d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 13:44:39.156315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.156287 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22efcf-6bccb7db6d-gwdxh"] Oct 30 13:44:39.820870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.820835 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:39.960659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:39.960604 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" podUID="6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:40.013730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:40.013684 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380c7464-dcec-4c9a-9bce-d54738f6073e" path="/var/lib/kubelet/pods/380c7464-dcec-4c9a-9bce-d54738f6073e/volumes" Oct 30 13:44:40.709125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:40.709096 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:40.709512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:40.709492 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:40.710355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:40.710336 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ljfbp" Oct 30 13:44:40.784537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:40.784496 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:40.784925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:40.784904 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:40.785793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:40.785773 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:44:41.511646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:41.511611 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:43.100306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:43.100237 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:44.254290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:44.253914 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:44.898194 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:44.898157 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:46.630144 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:46.630109 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:48.342630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:48.342594 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:49.501585 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:49.501550 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:49.960496 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:49.960455 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" podUID="6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:50.338424 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:50.338324 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:52.092789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:52.092754 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:53.798627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:53.798596 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:54.849138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:54.849106 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx"] Oct 30 13:44:54.887956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:54.887913 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf"] Oct 30 13:44:55.117854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:55.117760 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:55.325181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:55.325143 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:55.541380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:55.541326 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:56.628209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:56.628178 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:57.265845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:57.265780 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:58.279543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.279515 5515 generic.go:356] "Generic (PLEG): container finished" podID="6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" containerID="244cfd869ada591cb80cfb5a60affd32a4d67c95fd7b567f69c37c7ab06372cc" exitCode=0 Oct 30 13:44:58.279894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.279594 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" event={"ID":"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c","Type":"ContainerDied","Data":"244cfd869ada591cb80cfb5a60affd32a4d67c95fd7b567f69c37c7ab06372cc"} Oct 30 13:44:58.279894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.279659 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" event={"ID":"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c","Type":"ContainerDied","Data":"010a24f7deb97875761f149c1af927661d427aa304d5c6aa959260ccb4f07232"} Oct 30 13:44:58.279894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.279687 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="010a24f7deb97875761f149c1af927661d427aa304d5c6aa959260ccb4f07232" Oct 30 13:44:58.298207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.298187 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:44:58.331410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.331389 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-plugins-config-volume\") pod \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\" (UID: \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\") " Oct 30 13:44:58.331631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.331618 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-kube-api-access-ms5hk\" (UniqueName: \"kubernetes.io/projected/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-kube-api-access-ms5hk\") pod \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\" (UID: \"6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c\") " Oct 30 13:44:58.331736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.331717 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-plugins-config-volume" pod "6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" (UID: "6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:58.332183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.332162 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:44:58.333718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.333701 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-kube-api-access-ms5hk" pod "6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" (UID: "6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c"). InnerVolumeSpecName "kube-api-access-ms5hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:58.433615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:58.433583 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms5hk\" (UniqueName: \"kubernetes.io/projected/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c-kube-api-access-ms5hk\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:44:59.023350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:59.023317 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:44:59.032396 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:44:59.032361 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:44:59.287565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:59.287487 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm" Oct 30 13:44:59.342059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:59.342020 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm"] Oct 30 13:44:59.354947 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:59.354919 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22efcf-5cb4946c79-hjhfm"] Oct 30 13:44:59.849751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:59.849713 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" podUID="aff4dd5f-793d-4148-8be2-8cbe26c2eddf" containerName="payload-processing" containerID="cri-o://d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231" gracePeriod=25 Oct 30 13:44:59.889244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:44:59.889203 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" podUID="bf045354-c623-4b61-bbd9-ab57faeb18a2" containerName="payload-pre-processing" containerID="cri-o://b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13" gracePeriod=25 Oct 30 13:45:00.004254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.004219 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" path="/var/lib/kubelet/pods/6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c/volumes" Oct 30 13:45:00.310803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.310752 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh"] Oct 30 13:45:00.312029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.312001 5515 cpu_manager.go:414] "containerMap: removing container" podUID="6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" containerName="payload-pre-processing" Oct 30 13:45:00.312029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.312029 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" containerName="payload-pre-processing" Oct 30 13:45:00.312531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.312048 5515 cpu_manager.go:414] "containerMap: removing container" podUID="380c7464-dcec-4c9a-9bce-d54738f6073e" containerName="payload-processing" Oct 30 13:45:00.312531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.312056 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380c7464-dcec-4c9a-9bce-d54738f6073e" containerName="payload-processing" Oct 30 13:45:00.312531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.312371 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca90c36-dd12-47b0-a6d7-d01f0d79ab7c" containerName="payload-pre-processing" Oct 30 13:45:00.312531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.312391 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="380c7464-dcec-4c9a-9bce-d54738f6073e" containerName="payload-processing" Oct 30 13:45:00.321068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.321045 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:00.322616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.322591 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn"] Oct 30 13:45:00.330715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.330694 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:00.343288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.338785 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7"] Oct 30 13:45:00.343288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.342139 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\"" Oct 30 13:45:00.345671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.345649 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:00.346122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.346104 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh"] Oct 30 13:45:00.354029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.353825 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw"] Oct 30 13:45:00.363695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.363666 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:00.367218 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.367187 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43855476-25c9-4939-bfc5-3701cf7f04e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn\" (UID: \"43855476-25c9-4939-bfc5-3701cf7f04e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:00.367366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.367299 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp4jx\" (UniqueName: \"kubernetes.io/projected/43855476-25c9-4939-bfc5-3701cf7f04e9-kube-api-access-lp4jx\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn\" (UID: \"43855476-25c9-4939-bfc5-3701cf7f04e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:00.367442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.367421 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh\" (UID: \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:00.367518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.367501 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc6l7\" (UniqueName: \"kubernetes.io/projected/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-kube-api-access-zc6l7\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh\" (UID: \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:00.368175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.368155 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn"] Oct 30 13:45:00.380460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.380435 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7"] Oct 30 13:45:00.396742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.396716 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh"] Oct 30 13:45:00.424640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.424581 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw"] Oct 30 13:45:00.424836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.424682 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t"] Oct 30 13:45:00.425989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.424897 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:00.432906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.432882 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh"] Oct 30 13:45:00.433046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.432925 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t"] Oct 30 13:45:00.433046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.433009 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:00.484715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.484667 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmpzv\" (UniqueName: \"kubernetes.io/projected/78818086-4780-4d9b-be87-19b549bcdac2-kube-api-access-pmpzv\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t\" (UID: \"78818086-4780-4d9b-be87-19b549bcdac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:00.484982 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.484960 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgb4f\" (UniqueName: \"kubernetes.io/projected/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-kube-api-access-jgb4f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh\" (UID: \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:00.485153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.485135 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh\" (UID: \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:00.485303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.485286 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78818086-4780-4d9b-be87-19b549bcdac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t\" (UID: \"78818086-4780-4d9b-be87-19b549bcdac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:00.485507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.485491 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc6l7\" (UniqueName: \"kubernetes.io/projected/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-kube-api-access-zc6l7\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh\" (UID: \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:00.485657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.485642 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2w4h\" (UniqueName: \"kubernetes.io/projected/7c417014-5325-4155-838d-16a0e309477b-kube-api-access-z2w4h\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw\" (UID: \"7c417014-5325-4155-838d-16a0e309477b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:00.485926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.485909 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh\" (UID: \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:00.486102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.486086 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c417014-5325-4155-838d-16a0e309477b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw\" (UID: \"7c417014-5325-4155-838d-16a0e309477b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:00.486259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.486236 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43855476-25c9-4939-bfc5-3701cf7f04e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn\" (UID: \"43855476-25c9-4939-bfc5-3701cf7f04e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:00.486481 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.486464 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4jx\" (UniqueName: \"kubernetes.io/projected/43855476-25c9-4939-bfc5-3701cf7f04e9-kube-api-access-lp4jx\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn\" (UID: \"43855476-25c9-4939-bfc5-3701cf7f04e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:00.486653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.486625 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b49827-36bc-480e-b26e-82326bdc8b4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7\" (UID: \"92b49827-36bc-480e-b26e-82326bdc8b4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:00.487334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.487304 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cg7\" (UniqueName: \"kubernetes.io/projected/92b49827-36bc-480e-b26e-82326bdc8b4b-kube-api-access-q7cg7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7\" (UID: \"92b49827-36bc-480e-b26e-82326bdc8b4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:00.488325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.488301 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh\" (UID: \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:00.488583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.488554 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43855476-25c9-4939-bfc5-3701cf7f04e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn\" (UID: \"43855476-25c9-4939-bfc5-3701cf7f04e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:00.515070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.515027 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4jx\" (UniqueName: \"kubernetes.io/projected/43855476-25c9-4939-bfc5-3701cf7f04e9-kube-api-access-lp4jx\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn\" (UID: \"43855476-25c9-4939-bfc5-3701cf7f04e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:00.515950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.515930 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc6l7\" (UniqueName: \"kubernetes.io/projected/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-kube-api-access-zc6l7\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh\" (UID: \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:00.589671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.589582 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh\" (UID: \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:00.589671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.589642 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c417014-5325-4155-838d-16a0e309477b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw\" (UID: \"7c417014-5325-4155-838d-16a0e309477b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:00.589890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.589720 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b49827-36bc-480e-b26e-82326bdc8b4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7\" (UID: \"92b49827-36bc-480e-b26e-82326bdc8b4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:00.589890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.589745 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cg7\" (UniqueName: \"kubernetes.io/projected/92b49827-36bc-480e-b26e-82326bdc8b4b-kube-api-access-q7cg7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7\" (UID: \"92b49827-36bc-480e-b26e-82326bdc8b4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:00.589890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.589824 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpzv\" (UniqueName: \"kubernetes.io/projected/78818086-4780-4d9b-be87-19b549bcdac2-kube-api-access-pmpzv\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t\" (UID: \"78818086-4780-4d9b-be87-19b549bcdac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:00.589890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.589840 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgb4f\" (UniqueName: \"kubernetes.io/projected/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-kube-api-access-jgb4f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh\" (UID: \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:00.589890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.589860 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78818086-4780-4d9b-be87-19b549bcdac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t\" (UID: \"78818086-4780-4d9b-be87-19b549bcdac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:00.590526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.589956 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2w4h\" (UniqueName: \"kubernetes.io/projected/7c417014-5325-4155-838d-16a0e309477b-kube-api-access-z2w4h\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw\" (UID: \"7c417014-5325-4155-838d-16a0e309477b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:00.590571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.590543 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c417014-5325-4155-838d-16a0e309477b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw\" (UID: \"7c417014-5325-4155-838d-16a0e309477b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:00.590571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.590560 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b49827-36bc-480e-b26e-82326bdc8b4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7\" (UID: \"92b49827-36bc-480e-b26e-82326bdc8b4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:00.590645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.590543 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh\" (UID: \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:00.590734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.590710 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78818086-4780-4d9b-be87-19b549bcdac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t\" (UID: \"78818086-4780-4d9b-be87-19b549bcdac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:00.626040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.626007 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2w4h\" (UniqueName: \"kubernetes.io/projected/7c417014-5325-4155-838d-16a0e309477b-kube-api-access-z2w4h\") pod \"maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw\" (UID: \"7c417014-5325-4155-838d-16a0e309477b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:00.626207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.626007 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmpzv\" (UniqueName: \"kubernetes.io/projected/78818086-4780-4d9b-be87-19b549bcdac2-kube-api-access-pmpzv\") pod \"maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t\" (UID: \"78818086-4780-4d9b-be87-19b549bcdac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:00.626207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.626072 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cg7\" (UniqueName: \"kubernetes.io/projected/92b49827-36bc-480e-b26e-82326bdc8b4b-kube-api-access-q7cg7\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7\" (UID: \"92b49827-36bc-480e-b26e-82326bdc8b4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:00.626207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.626098 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgb4f\" (UniqueName: \"kubernetes.io/projected/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-kube-api-access-jgb4f\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh\" (UID: \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:00.667677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.667636 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:00.681118 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.681088 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:00.687837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.687809 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:00.700354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.700327 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:00.772454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.762591 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:00.772454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.771939 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:00.835121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:00.833643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:01.150380 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:45:01.150347 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43855476_25c9_4939_bfc5_3701cf7f04e9.slice/crio-c9e383b8b65adaf8a461614206d782a07c371d1dde3401370994a3e6f54c51a3 WatchSource:0}: Error finding container c9e383b8b65adaf8a461614206d782a07c371d1dde3401370994a3e6f54c51a3: Status 404 returned error can't find the container with id c9e383b8b65adaf8a461614206d782a07c371d1dde3401370994a3e6f54c51a3 Oct 30 13:45:01.152699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.152668 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn"] Oct 30 13:45:01.186790 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:45:01.186680 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c417014_5325_4155_838d_16a0e309477b.slice/crio-25e50a7e5616d2fe593343a9fb210f67750de831be5b83dd29da8fe0661c299a WatchSource:0}: Error finding container 25e50a7e5616d2fe593343a9fb210f67750de831be5b83dd29da8fe0661c299a: Status 404 returned error can't find the container with id 25e50a7e5616d2fe593343a9fb210f67750de831be5b83dd29da8fe0661c299a Oct 30 13:45:01.187258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.187235 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw"] Oct 30 13:45:01.190366 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:45:01.190341 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a3e642_d0bb_4d0d_8c2b_fbff1bc79a9c.slice/crio-e8b52fa8ce4a91507325029ae1a8c4bde38304561293c0b9e93f9b40642cfd34 WatchSource:0}: Error finding container e8b52fa8ce4a91507325029ae1a8c4bde38304561293c0b9e93f9b40642cfd34: Status 404 returned error can't find the container with id e8b52fa8ce4a91507325029ae1a8c4bde38304561293c0b9e93f9b40642cfd34 Oct 30 13:45:01.197617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.197589 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt"] Oct 30 13:45:01.197897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.197869 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" containerID="cri-o://d0a1ee1142db26a0e7eef572f649ca8bb4e660ab60dd7bbe0f5dc117483616a4" gracePeriod=30 Oct 30 13:45:01.202181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.202147 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh"] Oct 30 13:45:01.233565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.233534 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh"] Oct 30 13:45:01.237801 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.237773 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t"] Oct 30 13:45:01.310098 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:45:01.310063 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8adc93_a6b2_4ffe_8c69_08c7ba07789f.slice/crio-f7343cdc19f216b5ea854f61cb8af280bad77e6015812beca0b85040cb1e394c WatchSource:0}: Error finding container f7343cdc19f216b5ea854f61cb8af280bad77e6015812beca0b85040cb1e394c: Status 404 returned error can't find the container with id f7343cdc19f216b5ea854f61cb8af280bad77e6015812beca0b85040cb1e394c Oct 30 13:45:01.318438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.318406 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" event={"ID":"7c417014-5325-4155-838d-16a0e309477b","Type":"ContainerStarted","Data":"25e50a7e5616d2fe593343a9fb210f67750de831be5b83dd29da8fe0661c299a"} Oct 30 13:45:01.320049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.320026 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" event={"ID":"43855476-25c9-4939-bfc5-3701cf7f04e9","Type":"ContainerStarted","Data":"c9e383b8b65adaf8a461614206d782a07c371d1dde3401370994a3e6f54c51a3"} Oct 30 13:45:01.322115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.322091 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" event={"ID":"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c","Type":"ContainerStarted","Data":"e8b52fa8ce4a91507325029ae1a8c4bde38304561293c0b9e93f9b40642cfd34"} Oct 30 13:45:01.344109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:01.344063 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7"] Oct 30 13:45:01.347905 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:45:01.347875 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b49827_36bc_480e_b26e_82326bdc8b4b.slice/crio-7c9ab71317c371b2cb8f1ac2223edca22c5ac6be2ccd15287f4fef4b19436918 WatchSource:0}: Error finding container 7c9ab71317c371b2cb8f1ac2223edca22c5ac6be2ccd15287f4fef4b19436918: Status 404 returned error can't find the container with id 7c9ab71317c371b2cb8f1ac2223edca22c5ac6be2ccd15287f4fef4b19436918 Oct 30 13:45:02.332009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.331982 5515 generic.go:356] "Generic (PLEG): container finished" podID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerID="090a1196fe945fb236094a1a1600e640bdab1812efda598cee127cd62ce14640" exitCode=60 Oct 30 13:45:02.332404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.332067 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" event={"ID":"43855476-25c9-4939-bfc5-3701cf7f04e9","Type":"ContainerDied","Data":"090a1196fe945fb236094a1a1600e640bdab1812efda598cee127cd62ce14640"} Oct 30 13:45:02.332447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.332399 5515 scope.go:122] "RemoveContainer" containerID="090a1196fe945fb236094a1a1600e640bdab1812efda598cee127cd62ce14640" Oct 30 13:45:02.333935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.333907 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" event={"ID":"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c","Type":"ContainerStarted","Data":"e556e877452079a341c863479fe856d1b1f7508900a216017bda8971bd76ac17"} Oct 30 13:45:02.335994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.335969 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" event={"ID":"7c417014-5325-4155-838d-16a0e309477b","Type":"ContainerStarted","Data":"c80d9ff19c324491c993f43c889ef1a2a78fc4269a2370aa311efa43fc859605"} Oct 30 13:45:02.337720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.337697 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" event={"ID":"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f","Type":"ContainerStarted","Data":"2fa26ffbd8f920248b9dc6f639d82af57c9b69fbd00c8c5a7d9db287fe87e404"} Oct 30 13:45:02.337811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.337746 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" event={"ID":"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f","Type":"ContainerStarted","Data":"f7343cdc19f216b5ea854f61cb8af280bad77e6015812beca0b85040cb1e394c"} Oct 30 13:45:02.339758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.339725 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" event={"ID":"78818086-4780-4d9b-be87-19b549bcdac2","Type":"ContainerStarted","Data":"421cfe3eb95c05f4a0e11d52215f60d7fa9752845cfbb48b40e1caa2819c182b"} Oct 30 13:45:02.340142 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.340046 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" event={"ID":"78818086-4780-4d9b-be87-19b549bcdac2","Type":"ContainerStarted","Data":"8cc1684d8a6470a58a6b3afd7eedd22b8636849f306de554b59dee0d1ab96806"} Oct 30 13:45:02.341584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.341560 5515 generic.go:356] "Generic (PLEG): container finished" podID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerID="f1d5c7c0799349708df063e192a7389b1578daf266d22d4a6c92c4863abbd5bc" exitCode=60 Oct 30 13:45:02.341665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.341601 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" event={"ID":"92b49827-36bc-480e-b26e-82326bdc8b4b","Type":"ContainerDied","Data":"f1d5c7c0799349708df063e192a7389b1578daf266d22d4a6c92c4863abbd5bc"} Oct 30 13:45:02.341665 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.341638 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" event={"ID":"92b49827-36bc-480e-b26e-82326bdc8b4b","Type":"ContainerStarted","Data":"7c9ab71317c371b2cb8f1ac2223edca22c5ac6be2ccd15287f4fef4b19436918"} Oct 30 13:45:02.341893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.341877 5515 scope.go:122] "RemoveContainer" containerID="f1d5c7c0799349708df063e192a7389b1578daf266d22d4a6c92c4863abbd5bc" Oct 30 13:45:02.425676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.425614 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" podStartSLOduration=2.425592119 podStartE2EDuration="2.425592119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.421415073 +0000 UTC m=+4320.548215970" watchObservedRunningTime="2026-10-30 13:45:02.425592119 +0000 UTC m=+4320.552393019" Oct 30 13:45:02.473688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.473638 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" podStartSLOduration=2.473623085 podStartE2EDuration="2.473623085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.469716926 +0000 UTC m=+4320.596517824" watchObservedRunningTime="2026-10-30 13:45:02.473623085 +0000 UTC m=+4320.600424017" Oct 30 13:45:02.581290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.576898 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" podStartSLOduration=2.576880166 podStartE2EDuration="2.576880166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.566238845 +0000 UTC m=+4320.693039743" watchObservedRunningTime="2026-10-30 13:45:02.576880166 +0000 UTC m=+4320.703681066" Oct 30 13:45:02.610339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.610288 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" podStartSLOduration=2.610251019 podStartE2EDuration="2.610251019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.606933467 +0000 UTC m=+4320.733734366" watchObservedRunningTime="2026-10-30 13:45:02.610251019 +0000 UTC m=+4320.737051985" Oct 30 13:45:02.626712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:02.626683 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:03.352172 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.352135 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" event={"ID":"43855476-25c9-4939-bfc5-3701cf7f04e9","Type":"ContainerStarted","Data":"77f4b21f282fc81db97a88e3b48dfe34e1f690508f022d9a6eeb80749a4ad68d"} Oct 30 13:45:03.354104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.354073 5515 generic.go:356] "Generic (PLEG): container finished" podID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerID="e556e877452079a341c863479fe856d1b1f7508900a216017bda8971bd76ac17" exitCode=60 Oct 30 13:45:03.354256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.354158 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" event={"ID":"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c","Type":"ContainerDied","Data":"e556e877452079a341c863479fe856d1b1f7508900a216017bda8971bd76ac17"} Oct 30 13:45:03.354428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.354412 5515 scope.go:122] "RemoveContainer" containerID="e556e877452079a341c863479fe856d1b1f7508900a216017bda8971bd76ac17" Oct 30 13:45:03.357923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.357898 5515 generic.go:356] "Generic (PLEG): container finished" podID="7c417014-5325-4155-838d-16a0e309477b" containerID="c80d9ff19c324491c993f43c889ef1a2a78fc4269a2370aa311efa43fc859605" exitCode=60 Oct 30 13:45:03.358046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.357990 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" event={"ID":"7c417014-5325-4155-838d-16a0e309477b","Type":"ContainerDied","Data":"c80d9ff19c324491c993f43c889ef1a2a78fc4269a2370aa311efa43fc859605"} Oct 30 13:45:03.358496 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.358237 5515 scope.go:122] "RemoveContainer" containerID="c80d9ff19c324491c993f43c889ef1a2a78fc4269a2370aa311efa43fc859605" Oct 30 13:45:03.359917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.359884 5515 generic.go:356] "Generic (PLEG): container finished" podID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerID="2fa26ffbd8f920248b9dc6f639d82af57c9b69fbd00c8c5a7d9db287fe87e404" exitCode=60 Oct 30 13:45:03.359996 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.359956 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" event={"ID":"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f","Type":"ContainerDied","Data":"2fa26ffbd8f920248b9dc6f639d82af57c9b69fbd00c8c5a7d9db287fe87e404"} Oct 30 13:45:03.360215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.360195 5515 scope.go:122] "RemoveContainer" containerID="2fa26ffbd8f920248b9dc6f639d82af57c9b69fbd00c8c5a7d9db287fe87e404" Oct 30 13:45:03.361697 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.361678 5515 generic.go:356] "Generic (PLEG): container finished" podID="78818086-4780-4d9b-be87-19b549bcdac2" containerID="421cfe3eb95c05f4a0e11d52215f60d7fa9752845cfbb48b40e1caa2819c182b" exitCode=60 Oct 30 13:45:03.361807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.361779 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" event={"ID":"78818086-4780-4d9b-be87-19b549bcdac2","Type":"ContainerDied","Data":"421cfe3eb95c05f4a0e11d52215f60d7fa9752845cfbb48b40e1caa2819c182b"} Oct 30 13:45:03.361956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.361940 5515 scope.go:122] "RemoveContainer" containerID="421cfe3eb95c05f4a0e11d52215f60d7fa9752845cfbb48b40e1caa2819c182b" Oct 30 13:45:03.368107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.368076 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" event={"ID":"92b49827-36bc-480e-b26e-82326bdc8b4b","Type":"ContainerStarted","Data":"8718001b14e37030d882e507688895adbab6f2f24edb8af0d9864a3bac5aa944"} Oct 30 13:45:03.399598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.399542 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" podStartSLOduration=3.399522042 podStartE2EDuration="3.399522042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.395564102 +0000 UTC m=+4321.522365046" watchObservedRunningTime="2026-10-30 13:45:03.399522042 +0000 UTC m=+4321.526322940" Oct 30 13:45:03.602361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:03.602282 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" podStartSLOduration=3.602255315 podStartE2EDuration="3.602255315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.599397077 +0000 UTC m=+4321.726197987" watchObservedRunningTime="2026-10-30 13:45:03.602255315 +0000 UTC m=+4321.729056263" Oct 30 13:45:04.300738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.299645 5515 scope.go:122] "RemoveContainer" containerID="244cfd869ada591cb80cfb5a60affd32a4d67c95fd7b567f69c37c7ab06372cc" Oct 30 13:45:04.334936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.334191 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:04.377527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.377494 5515 generic.go:356] "Generic (PLEG): container finished" podID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerID="8718001b14e37030d882e507688895adbab6f2f24edb8af0d9864a3bac5aa944" exitCode=60 Oct 30 13:45:04.377937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.377578 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" event={"ID":"92b49827-36bc-480e-b26e-82326bdc8b4b","Type":"ContainerDied","Data":"8718001b14e37030d882e507688895adbab6f2f24edb8af0d9864a3bac5aa944"} Oct 30 13:45:04.377937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.377637 5515 scope.go:122] "RemoveContainer" containerID="f1d5c7c0799349708df063e192a7389b1578daf266d22d4a6c92c4863abbd5bc" Oct 30 13:45:04.378024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.377941 5515 scope.go:122] "RemoveContainer" containerID="8718001b14e37030d882e507688895adbab6f2f24edb8af0d9864a3bac5aa944" Oct 30 13:45:04.378336 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:04.378317 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7_odh-ai-gateway-infra(92b49827-36bc-480e-b26e-82326bdc8b4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" Oct 30 13:45:04.380203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.380184 5515 generic.go:356] "Generic (PLEG): container finished" podID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerID="77f4b21f282fc81db97a88e3b48dfe34e1f690508f022d9a6eeb80749a4ad68d" exitCode=60 Oct 30 13:45:04.380349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.380280 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" event={"ID":"43855476-25c9-4939-bfc5-3701cf7f04e9","Type":"ContainerDied","Data":"77f4b21f282fc81db97a88e3b48dfe34e1f690508f022d9a6eeb80749a4ad68d"} Oct 30 13:45:04.380574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.380559 5515 scope.go:122] "RemoveContainer" containerID="77f4b21f282fc81db97a88e3b48dfe34e1f690508f022d9a6eeb80749a4ad68d" Oct 30 13:45:04.380924 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:04.380902 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn_odh-ai-gateway-infra(43855476-25c9-4939-bfc5-3701cf7f04e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" Oct 30 13:45:04.382186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.382164 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" event={"ID":"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c","Type":"ContainerStarted","Data":"8ceda93c845abdb32d802bb791422efb51cf31a3491067fbe02f46f0261fd833"} Oct 30 13:45:04.384322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.384292 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" event={"ID":"7c417014-5325-4155-838d-16a0e309477b","Type":"ContainerStarted","Data":"b7afa6a8e1fc762f42fb0aadb75da0b164fc03beb4cccb69a1d389a074f78741"} Oct 30 13:45:04.386141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.386121 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" event={"ID":"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f","Type":"ContainerStarted","Data":"ad7bab388830e9bce7b8e038cd844fa08e7b5bccd44a57fa7dd1a81a24d2a4a3"} Oct 30 13:45:04.388019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.387994 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" event={"ID":"78818086-4780-4d9b-be87-19b549bcdac2","Type":"ContainerStarted","Data":"f8afa2a238d90d61258384360dc63f2b423f7dee54c7da108efa50df5a359017"} Oct 30 13:45:04.394254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:04.394230 5515 scope.go:122] "RemoveContainer" containerID="090a1196fe945fb236094a1a1600e640bdab1812efda598cee127cd62ce14640" Oct 30 13:45:05.402630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.402604 5515 scope.go:122] "RemoveContainer" containerID="77f4b21f282fc81db97a88e3b48dfe34e1f690508f022d9a6eeb80749a4ad68d" Oct 30 13:45:05.403104 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:05.403025 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn_odh-ai-gateway-infra(43855476-25c9-4939-bfc5-3701cf7f04e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" Oct 30 13:45:05.404752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.404722 5515 generic.go:356] "Generic (PLEG): container finished" podID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerID="8ceda93c845abdb32d802bb791422efb51cf31a3491067fbe02f46f0261fd833" exitCode=60 Oct 30 13:45:05.404877 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.404817 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" event={"ID":"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c","Type":"ContainerDied","Data":"8ceda93c845abdb32d802bb791422efb51cf31a3491067fbe02f46f0261fd833"} Oct 30 13:45:05.404939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.404917 5515 scope.go:122] "RemoveContainer" containerID="e556e877452079a341c863479fe856d1b1f7508900a216017bda8971bd76ac17" Oct 30 13:45:05.405089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.405067 5515 scope.go:122] "RemoveContainer" containerID="8ceda93c845abdb32d802bb791422efb51cf31a3491067fbe02f46f0261fd833" Oct 30 13:45:05.405507 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:05.405479 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh_odh-ai-gateway-infra(95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" Oct 30 13:45:05.406973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.406948 5515 generic.go:356] "Generic (PLEG): container finished" podID="7c417014-5325-4155-838d-16a0e309477b" containerID="b7afa6a8e1fc762f42fb0aadb75da0b164fc03beb4cccb69a1d389a074f78741" exitCode=60 Oct 30 13:45:05.407076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.407024 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" event={"ID":"7c417014-5325-4155-838d-16a0e309477b","Type":"ContainerDied","Data":"b7afa6a8e1fc762f42fb0aadb75da0b164fc03beb4cccb69a1d389a074f78741"} Oct 30 13:45:05.407391 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.407373 5515 scope.go:122] "RemoveContainer" containerID="b7afa6a8e1fc762f42fb0aadb75da0b164fc03beb4cccb69a1d389a074f78741" Oct 30 13:45:05.407738 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:05.407715 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw_odh-ai-gateway-infra(7c417014-5325-4155-838d-16a0e309477b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" podUID="7c417014-5325-4155-838d-16a0e309477b" Oct 30 13:45:05.409180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.409160 5515 generic.go:356] "Generic (PLEG): container finished" podID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerID="ad7bab388830e9bce7b8e038cd844fa08e7b5bccd44a57fa7dd1a81a24d2a4a3" exitCode=60 Oct 30 13:45:05.409298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.409243 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" event={"ID":"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f","Type":"ContainerDied","Data":"ad7bab388830e9bce7b8e038cd844fa08e7b5bccd44a57fa7dd1a81a24d2a4a3"} Oct 30 13:45:05.409499 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.409482 5515 scope.go:122] "RemoveContainer" containerID="ad7bab388830e9bce7b8e038cd844fa08e7b5bccd44a57fa7dd1a81a24d2a4a3" Oct 30 13:45:05.409812 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:05.409782 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh_odh-ai-gateway-infra(1d8adc93-a6b2-4ffe-8c69-08c7ba07789f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" Oct 30 13:45:05.411335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.411311 5515 generic.go:356] "Generic (PLEG): container finished" podID="78818086-4780-4d9b-be87-19b549bcdac2" containerID="f8afa2a238d90d61258384360dc63f2b423f7dee54c7da108efa50df5a359017" exitCode=60 Oct 30 13:45:05.411427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.411345 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" event={"ID":"78818086-4780-4d9b-be87-19b549bcdac2","Type":"ContainerDied","Data":"f8afa2a238d90d61258384360dc63f2b423f7dee54c7da108efa50df5a359017"} Oct 30 13:45:05.411641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.411624 5515 scope.go:122] "RemoveContainer" containerID="f8afa2a238d90d61258384360dc63f2b423f7dee54c7da108efa50df5a359017" Oct 30 13:45:05.412111 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:05.411919 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t_odh-ai-gateway-infra(78818086-4780-4d9b-be87-19b549bcdac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" podUID="78818086-4780-4d9b-be87-19b549bcdac2" Oct 30 13:45:05.413515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.413496 5515 scope.go:122] "RemoveContainer" containerID="8718001b14e37030d882e507688895adbab6f2f24edb8af0d9864a3bac5aa944" Oct 30 13:45:05.413868 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:05.413812 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7_odh-ai-gateway-infra(92b49827-36bc-480e-b26e-82326bdc8b4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" Oct 30 13:45:05.420139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.420123 5515 scope.go:122] "RemoveContainer" containerID="c80d9ff19c324491c993f43c889ef1a2a78fc4269a2370aa311efa43fc859605" Oct 30 13:45:05.434344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.434327 5515 scope.go:122] "RemoveContainer" containerID="2fa26ffbd8f920248b9dc6f639d82af57c9b69fbd00c8c5a7d9db287fe87e404" Oct 30 13:45:05.449537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:05.449517 5515 scope.go:122] "RemoveContainer" containerID="421cfe3eb95c05f4a0e11d52215f60d7fa9752845cfbb48b40e1caa2819c182b" Oct 30 13:45:06.136598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.136567 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:06.425322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.425227 5515 generic.go:356] "Generic (PLEG): container finished" podID="7252b9af-d947-446c-bf46-0d6802a57e12" containerID="d0a1ee1142db26a0e7eef572f649ca8bb4e660ab60dd7bbe0f5dc117483616a4" exitCode=0 Oct 30 13:45:06.425803 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.425307 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" event={"ID":"7252b9af-d947-446c-bf46-0d6802a57e12","Type":"ContainerDied","Data":"d0a1ee1142db26a0e7eef572f649ca8bb4e660ab60dd7bbe0f5dc117483616a4"} Oct 30 13:45:06.427843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.427817 5515 scope.go:122] "RemoveContainer" containerID="8ceda93c845abdb32d802bb791422efb51cf31a3491067fbe02f46f0261fd833" Oct 30 13:45:06.428385 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:06.428343 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh_odh-ai-gateway-infra(95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" Oct 30 13:45:06.429973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.429945 5515 scope.go:122] "RemoveContainer" containerID="b7afa6a8e1fc762f42fb0aadb75da0b164fc03beb4cccb69a1d389a074f78741" Oct 30 13:45:06.430400 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:06.430373 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw_odh-ai-gateway-infra(7c417014-5325-4155-838d-16a0e309477b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" podUID="7c417014-5325-4155-838d-16a0e309477b" Oct 30 13:45:06.432667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.432644 5515 scope.go:122] "RemoveContainer" containerID="ad7bab388830e9bce7b8e038cd844fa08e7b5bccd44a57fa7dd1a81a24d2a4a3" Oct 30 13:45:06.433033 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:06.433008 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh_odh-ai-gateway-infra(1d8adc93-a6b2-4ffe-8c69-08c7ba07789f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" Oct 30 13:45:06.435101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.435080 5515 scope.go:122] "RemoveContainer" containerID="f8afa2a238d90d61258384360dc63f2b423f7dee54c7da108efa50df5a359017" Oct 30 13:45:06.435392 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:06.435360 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t_odh-ai-gateway-infra(78818086-4780-4d9b-be87-19b549bcdac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" podUID="78818086-4780-4d9b-be87-19b549bcdac2" Oct 30 13:45:06.489104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.489081 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:45:06.599713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.599677 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-envoy\") pod \"7252b9af-d947-446c-bf46-0d6802a57e12\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " Oct 30 13:45:06.599912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.599744 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-certs\") pod \"7252b9af-d947-446c-bf46-0d6802a57e12\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " Oct 30 13:45:06.599912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.599782 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-kube-api-access-kfwlb\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-kube-api-access-kfwlb\") pod \"7252b9af-d947-446c-bf46-0d6802a57e12\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " Oct 30 13:45:06.599912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.599822 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-istio-token\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-istio-token\") pod \"7252b9af-d947-446c-bf46-0d6802a57e12\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " Oct 30 13:45:06.600096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.599986 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-socket\") pod \"7252b9af-d947-446c-bf46-0d6802a57e12\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " Oct 30 13:45:06.600096 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.600088 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-data\") pod \"7252b9af-d947-446c-bf46-0d6802a57e12\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " Oct 30 13:45:06.600211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.600125 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7252b9af-d947-446c-bf46-0d6802a57e12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7252b9af-d947-446c-bf46-0d6802a57e12-istio-podinfo\") pod \"7252b9af-d947-446c-bf46-0d6802a57e12\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " Oct 30 13:45:06.600288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.600212 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7252b9af-d947-446c-bf46-0d6802a57e12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7252b9af-d947-446c-bf46-0d6802a57e12-istiod-ca-cert\") pod \"7252b9af-d947-446c-bf46-0d6802a57e12\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " Oct 30 13:45:06.600609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.600582 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7252b9af-d947-446c-bf46-0d6802a57e12-istiod-ca-cert" pod "7252b9af-d947-446c-bf46-0d6802a57e12" (UID: "7252b9af-d947-446c-bf46-0d6802a57e12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:06.600814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.600797 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-socket" pod "7252b9af-d947-446c-bf46-0d6802a57e12" (UID: "7252b9af-d947-446c-bf46-0d6802a57e12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:06.600978 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.600961 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-data" pod "7252b9af-d947-446c-bf46-0d6802a57e12" (UID: "7252b9af-d947-446c-bf46-0d6802a57e12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:06.601580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.601386 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-credential-socket\") pod \"7252b9af-d947-446c-bf46-0d6802a57e12\" (UID: \"7252b9af-d947-446c-bf46-0d6802a57e12\") " Oct 30 13:45:06.602124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.602093 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-certs" pod "7252b9af-d947-446c-bf46-0d6802a57e12" (UID: "7252b9af-d947-446c-bf46-0d6802a57e12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:06.602347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.602324 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-credential-socket" pod "7252b9af-d947-446c-bf46-0d6802a57e12" (UID: "7252b9af-d947-446c-bf46-0d6802a57e12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:06.602827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.602801 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.602921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.602833 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.602921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.602847 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7252b9af-d947-446c-bf46-0d6802a57e12-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.602921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.602861 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.602921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.602873 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.604139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.604090 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-envoy" pod "7252b9af-d947-446c-bf46-0d6802a57e12" (UID: "7252b9af-d947-446c-bf46-0d6802a57e12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:06.604250 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.604203 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-istio-token" pod "7252b9af-d947-446c-bf46-0d6802a57e12" (UID: "7252b9af-d947-446c-bf46-0d6802a57e12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:06.604365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.604329 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7252b9af-d947-446c-bf46-0d6802a57e12-istio-podinfo" pod "7252b9af-d947-446c-bf46-0d6802a57e12" (UID: "7252b9af-d947-446c-bf46-0d6802a57e12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:06.605688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.605668 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-kube-api-access-kfwlb" pod "7252b9af-d947-446c-bf46-0d6802a57e12" (UID: "7252b9af-d947-446c-bf46-0d6802a57e12"). InnerVolumeSpecName "kube-api-access-kfwlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:06.704614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.704533 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7252b9af-d947-446c-bf46-0d6802a57e12-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.704614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.704564 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfwlb\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-kube-api-access-kfwlb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.704614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.704577 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7252b9af-d947-446c-bf46-0d6802a57e12-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:06.704614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:06.704591 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7252b9af-d947-446c-bf46-0d6802a57e12-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:07.053124 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:07.032348 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:07.446279 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:07.446251 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" Oct 30 13:45:07.446697 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:07.446313 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt" event={"ID":"7252b9af-d947-446c-bf46-0d6802a57e12","Type":"ContainerDied","Data":"9e581748aa4eddd21ae23d461b8767b3ba5069fb6ebfd73776888915eb6da162"} Oct 30 13:45:07.446697 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:07.446373 5515 scope.go:122] "RemoveContainer" containerID="d0a1ee1142db26a0e7eef572f649ca8bb4e660ab60dd7bbe0f5dc117483616a4" Oct 30 13:45:07.568133 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:07.568054 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt"] Oct 30 13:45:07.584184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:07.584007 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-v76vt"] Oct 30 13:45:07.732347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:07.727838 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:08.004865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.004788 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" path="/var/lib/kubelet/pods/7252b9af-d947-446c-bf46-0d6802a57e12/volumes" Oct 30 13:45:08.135198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.135164 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25"] Oct 30 13:45:08.136336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.136204 5515 cpu_manager.go:414] "containerMap: removing container" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" Oct 30 13:45:08.136336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.136227 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" Oct 30 13:45:08.136518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.136508 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7252b9af-d947-446c-bf46-0d6802a57e12" containerName="istio-proxy" Oct 30 13:45:08.142311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.142286 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.175550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.174617 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:45:08.187312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.185134 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25"] Oct 30 13:45:08.215951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.215915 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:08.245483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.245442 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/a6e53cfa-6c99-4543-84ae-be9dbadde34e-kube-api-access-vnhgp\") pod \"maas-api-revoke-sub-f6fbab4a-4kw25\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.245703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.245591 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fbab4a-4kw25\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.245782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.245743 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fbab4a-4kw25\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.347217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.347122 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fbab4a-4kw25\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.347418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.347379 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/a6e53cfa-6c99-4543-84ae-be9dbadde34e-kube-api-access-vnhgp\") pod \"maas-api-revoke-sub-f6fbab4a-4kw25\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.347516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.347498 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fbab4a-4kw25\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.347866 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.347845 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fbab4a-4kw25\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.350499 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.350471 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fbab4a-4kw25\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.371322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.371252 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/a6e53cfa-6c99-4543-84ae-be9dbadde34e-kube-api-access-vnhgp\") pod \"maas-api-revoke-sub-f6fbab4a-4kw25\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.486137 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.486109 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:08.646661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.646613 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" podUID="bf045354-c623-4b61-bbd9-ab57faeb18a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:08.646874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.646741 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" podUID="aff4dd5f-793d-4148-8be2-8cbe26c2eddf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:08.795455 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:45:08.795428 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e53cfa_6c99_4543_84ae_be9dbadde34e.slice/crio-fa66782630d2e146b77f706c8a6255e7407d1bd0d27797d4c6cfd4e0840c64a4 WatchSource:0}: Error finding container fa66782630d2e146b77f706c8a6255e7407d1bd0d27797d4c6cfd4e0840c64a4: Status 404 returned error can't find the container with id fa66782630d2e146b77f706c8a6255e7407d1bd0d27797d4c6cfd4e0840c64a4 Oct 30 13:45:08.798716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:08.798509 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25"] Oct 30 13:45:09.466410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:09.466364 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" event={"ID":"a6e53cfa-6c99-4543-84ae-be9dbadde34e","Type":"ContainerStarted","Data":"291bc29e9d5eab5324ea5bbfdd56ab9b7204527a90b1f08848afe9c8a87db58a"} Oct 30 13:45:09.466593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:09.466436 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" event={"ID":"a6e53cfa-6c99-4543-84ae-be9dbadde34e","Type":"ContainerStarted","Data":"fa66782630d2e146b77f706c8a6255e7407d1bd0d27797d4c6cfd4e0840c64a4"} Oct 30 13:45:09.516869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:09.516809 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" podStartSLOduration=1.516795645 podStartE2EDuration="1.516795645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:09.51249199 +0000 UTC m=+4327.639292888" watchObservedRunningTime="2026-10-30 13:45:09.516795645 +0000 UTC m=+4327.643596544" Oct 30 13:45:09.952164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:09.952133 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:10.482049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:10.482006 5515 generic.go:356] "Generic (PLEG): container finished" podID="a6e53cfa-6c99-4543-84ae-be9dbadde34e" containerID="291bc29e9d5eab5324ea5bbfdd56ab9b7204527a90b1f08848afe9c8a87db58a" exitCode=0 Oct 30 13:45:10.482233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:10.482050 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" event={"ID":"a6e53cfa-6c99-4543-84ae-be9dbadde34e","Type":"ContainerDied","Data":"291bc29e9d5eab5324ea5bbfdd56ab9b7204527a90b1f08848afe9c8a87db58a"} Oct 30 13:45:11.689440 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.689409 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:11.692062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.692030 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:11.799610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.799523 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-openshift-ca\") pod \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " Oct 30 13:45:11.799774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.799673 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e53cfa-6c99-4543-84ae-be9dbadde34e-kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/a6e53cfa-6c99-4543-84ae-be9dbadde34e-kube-api-access-vnhgp\") pod \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " Oct 30 13:45:11.799855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.799811 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-xks-ca\") pod \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\" (UID: \"a6e53cfa-6c99-4543-84ae-be9dbadde34e\") " Oct 30 13:45:11.799948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.799928 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-openshift-ca" pod "a6e53cfa-6c99-4543-84ae-be9dbadde34e" (UID: "a6e53cfa-6c99-4543-84ae-be9dbadde34e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:11.800707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.800674 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-openshift-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:11.802011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.801979 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-xks-ca" pod "a6e53cfa-6c99-4543-84ae-be9dbadde34e" (UID: "a6e53cfa-6c99-4543-84ae-be9dbadde34e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:11.802165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.802132 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e53cfa-6c99-4543-84ae-be9dbadde34e-kube-api-access-vnhgp" pod "a6e53cfa-6c99-4543-84ae-be9dbadde34e" (UID: "a6e53cfa-6c99-4543-84ae-be9dbadde34e"). InnerVolumeSpecName "kube-api-access-vnhgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:11.902398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.902363 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/a6e53cfa-6c99-4543-84ae-be9dbadde34e-kube-api-access-vnhgp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:11.902398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:11.902393 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e53cfa-6c99-4543-84ae-be9dbadde34e-maas-api-xks-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:12.500164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:12.500130 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" Oct 30 13:45:12.500354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:12.500164 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25" event={"ID":"a6e53cfa-6c99-4543-84ae-be9dbadde34e","Type":"ContainerDied","Data":"fa66782630d2e146b77f706c8a6255e7407d1bd0d27797d4c6cfd4e0840c64a4"} Oct 30 13:45:12.500354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:12.500214 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa66782630d2e146b77f706c8a6255e7407d1bd0d27797d4c6cfd4e0840c64a4" Oct 30 13:45:13.430926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:13.430869 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:14.513650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:14.513611 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:45:14.972798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:14.972768 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:15.587544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:15.587503 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:15.800662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:15.800633 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:16.003919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:16.003887 5515 scope.go:122] "RemoveContainer" containerID="ad7bab388830e9bce7b8e038cd844fa08e7b5bccd44a57fa7dd1a81a24d2a4a3" Oct 30 13:45:16.040570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:16.040530 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:16.271362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:16.271287 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:17.011342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:17.005555 5515 scope.go:122] "RemoveContainer" containerID="8718001b14e37030d882e507688895adbab6f2f24edb8af0d9864a3bac5aa944" Oct 30 13:45:17.011342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:17.006298 5515 scope.go:122] "RemoveContainer" containerID="77f4b21f282fc81db97a88e3b48dfe34e1f690508f022d9a6eeb80749a4ad68d" Oct 30 13:45:17.042390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:17.042364 5515 scope.go:122] "RemoveContainer" containerID="f8afa2a238d90d61258384360dc63f2b423f7dee54c7da108efa50df5a359017" Oct 30 13:45:17.599324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:17.596812 5515 generic.go:356] "Generic (PLEG): container finished" podID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerID="3c7020f38f121f32082bbdbd691bcc57b9a040d79bcb73be5e10c93b88489a65" exitCode=60 Oct 30 13:45:17.599324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:17.597135 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" event={"ID":"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f","Type":"ContainerDied","Data":"3c7020f38f121f32082bbdbd691bcc57b9a040d79bcb73be5e10c93b88489a65"} Oct 30 13:45:17.599324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:17.597198 5515 scope.go:122] "RemoveContainer" containerID="ad7bab388830e9bce7b8e038cd844fa08e7b5bccd44a57fa7dd1a81a24d2a4a3" Oct 30 13:45:17.599324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:17.597789 5515 scope.go:122] "RemoveContainer" containerID="3c7020f38f121f32082bbdbd691bcc57b9a040d79bcb73be5e10c93b88489a65" Oct 30 13:45:17.599324 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:17.598404 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh_odh-ai-gateway-infra(1d8adc93-a6b2-4ffe-8c69-08c7ba07789f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" Oct 30 13:45:17.966314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:17.966280 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:18.684617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.684523 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" podUID="bf045354-c623-4b61-bbd9-ab57faeb18a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:18.685641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.685602 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" podUID="aff4dd5f-793d-4148-8be2-8cbe26c2eddf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:45:18.687332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.687296 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh"] Oct 30 13:45:18.697720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.697670 5515 generic.go:356] "Generic (PLEG): container finished" podID="78818086-4780-4d9b-be87-19b549bcdac2" containerID="dc79573d074002af5351b0e59458b9c06475ffb93cfb7c07a4b08577bd9d07be" exitCode=60 Oct 30 13:45:18.698711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.697863 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" event={"ID":"78818086-4780-4d9b-be87-19b549bcdac2","Type":"ContainerDied","Data":"dc79573d074002af5351b0e59458b9c06475ffb93cfb7c07a4b08577bd9d07be"} Oct 30 13:45:18.698711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.697944 5515 scope.go:122] "RemoveContainer" containerID="f8afa2a238d90d61258384360dc63f2b423f7dee54c7da108efa50df5a359017" Oct 30 13:45:18.698711 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.698571 5515 scope.go:122] "RemoveContainer" containerID="dc79573d074002af5351b0e59458b9c06475ffb93cfb7c07a4b08577bd9d07be" Oct 30 13:45:18.699292 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:18.699079 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t_odh-ai-gateway-infra(78818086-4780-4d9b-be87-19b549bcdac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" podUID="78818086-4780-4d9b-be87-19b549bcdac2" Oct 30 13:45:18.716772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.716742 5515 generic.go:356] "Generic (PLEG): container finished" podID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerID="72097f4cb87c7062136503fe36ba1008243be33d308351cbbf3133066b2008a8" exitCode=60 Oct 30 13:45:18.717333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.717283 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" event={"ID":"92b49827-36bc-480e-b26e-82326bdc8b4b","Type":"ContainerDied","Data":"72097f4cb87c7062136503fe36ba1008243be33d308351cbbf3133066b2008a8"} Oct 30 13:45:18.717983 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.717945 5515 scope.go:122] "RemoveContainer" containerID="72097f4cb87c7062136503fe36ba1008243be33d308351cbbf3133066b2008a8" Oct 30 13:45:18.718710 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:18.718682 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7_odh-ai-gateway-infra(92b49827-36bc-480e-b26e-82326bdc8b4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" Oct 30 13:45:18.721074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.721052 5515 generic.go:356] "Generic (PLEG): container finished" podID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerID="81b153f4ed741c6a175830432023edb770893c6e8e1c7009e05b2f2ac292059b" exitCode=60 Oct 30 13:45:18.721305 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.721289 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" event={"ID":"43855476-25c9-4939-bfc5-3701cf7f04e9","Type":"ContainerDied","Data":"81b153f4ed741c6a175830432023edb770893c6e8e1c7009e05b2f2ac292059b"} Oct 30 13:45:18.721822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.721806 5515 scope.go:122] "RemoveContainer" containerID="81b153f4ed741c6a175830432023edb770893c6e8e1c7009e05b2f2ac292059b" Oct 30 13:45:18.722385 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:18.722327 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn_odh-ai-gateway-infra(43855476-25c9-4939-bfc5-3701cf7f04e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" Oct 30 13:45:18.780796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.780415 5515 scope.go:122] "RemoveContainer" containerID="8718001b14e37030d882e507688895adbab6f2f24edb8af0d9864a3bac5aa944" Oct 30 13:45:18.804720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.804216 5515 scope.go:122] "RemoveContainer" containerID="77f4b21f282fc81db97a88e3b48dfe34e1f690508f022d9a6eeb80749a4ad68d" Oct 30 13:45:19.005239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.994754 5515 scope.go:122] "RemoveContainer" containerID="b7afa6a8e1fc762f42fb0aadb75da0b164fc03beb4cccb69a1d389a074f78741" Oct 30 13:45:19.005239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:18.996992 5515 scope.go:122] "RemoveContainer" containerID="8ceda93c845abdb32d802bb791422efb51cf31a3491067fbe02f46f0261fd833" Oct 30 13:45:19.180814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.180355 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:19.292005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.291907 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-kube-api-access-jgb4f\" (UniqueName: \"kubernetes.io/projected/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-kube-api-access-jgb4f\") pod \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\" (UID: \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\") " Oct 30 13:45:19.292192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.292101 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-maas-api-service-ca\") pod \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\" (UID: \"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f\") " Oct 30 13:45:19.292664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.292621 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-maas-api-service-ca" pod "1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" (UID: "1d8adc93-a6b2-4ffe-8c69-08c7ba07789f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:19.293166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.293140 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:19.359322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.341038 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-kube-api-access-jgb4f" pod "1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" (UID: "1d8adc93-a6b2-4ffe-8c69-08c7ba07789f"). InnerVolumeSpecName "kube-api-access-jgb4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:19.394687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.394651 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgb4f\" (UniqueName: \"kubernetes.io/projected/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f-kube-api-access-jgb4f\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:19.738408 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.738371 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" event={"ID":"7c417014-5325-4155-838d-16a0e309477b","Type":"ContainerStarted","Data":"5ab6e68ce7164437c54ca663ad69e4cd10970fb35b6231e1e1d88c4429efa95c"} Oct 30 13:45:19.741253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.741221 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" event={"ID":"1d8adc93-a6b2-4ffe-8c69-08c7ba07789f","Type":"ContainerDied","Data":"f7343cdc19f216b5ea854f61cb8af280bad77e6015812beca0b85040cb1e394c"} Oct 30 13:45:19.741414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.741304 5515 scope.go:122] "RemoveContainer" containerID="3c7020f38f121f32082bbdbd691bcc57b9a040d79bcb73be5e10c93b88489a65" Oct 30 13:45:19.741414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.741388 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh" Oct 30 13:45:19.882299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.882245 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh"] Oct 30 13:45:19.887346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.887320 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t"] Oct 30 13:45:19.891975 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.891949 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889465-ct8dh"] Oct 30 13:45:19.907200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.906928 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn"] Oct 30 13:45:19.922008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:19.921975 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7"] Oct 30 13:45:20.092403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.092344 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" path="/var/lib/kubelet/pods/1d8adc93-a6b2-4ffe-8c69-08c7ba07789f/volumes" Oct 30 13:45:20.116608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.116577 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:20.246489 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.246465 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:20.357145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.353522 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43855476-25c9-4939-bfc5-3701cf7f04e9-kube-api-access-lp4jx\" (UniqueName: \"kubernetes.io/projected/43855476-25c9-4939-bfc5-3701cf7f04e9-kube-api-access-lp4jx\") pod \"43855476-25c9-4939-bfc5-3701cf7f04e9\" (UID: \"43855476-25c9-4939-bfc5-3701cf7f04e9\") " Oct 30 13:45:20.357145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.353580 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43855476-25c9-4939-bfc5-3701cf7f04e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43855476-25c9-4939-bfc5-3701cf7f04e9-maas-api-service-ca\") pod \"43855476-25c9-4939-bfc5-3701cf7f04e9\" (UID: \"43855476-25c9-4939-bfc5-3701cf7f04e9\") " Oct 30 13:45:20.383843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.383805 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43855476-25c9-4939-bfc5-3701cf7f04e9-kube-api-access-lp4jx" pod "43855476-25c9-4939-bfc5-3701cf7f04e9" (UID: "43855476-25c9-4939-bfc5-3701cf7f04e9"). InnerVolumeSpecName "kube-api-access-lp4jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:20.384224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.384191 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43855476-25c9-4939-bfc5-3701cf7f04e9-maas-api-service-ca" pod "43855476-25c9-4939-bfc5-3701cf7f04e9" (UID: "43855476-25c9-4939-bfc5-3701cf7f04e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:20.426138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.425927 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:20.463211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.456633 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp4jx\" (UniqueName: \"kubernetes.io/projected/43855476-25c9-4939-bfc5-3701cf7f04e9-kube-api-access-lp4jx\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.463211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.456833 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43855476-25c9-4939-bfc5-3701cf7f04e9-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.497232 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5"] Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.498806 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.498827 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.498850 5515 cpu_manager.go:414] "containerMap: removing container" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.498859 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.498964 5515 cpu_manager.go:414] "containerMap: removing container" podUID="a6e53cfa-6c99-4543-84ae-be9dbadde34e" containerName="revoke-keys" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.498974 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e53cfa-6c99-4543-84ae-be9dbadde34e" containerName="revoke-keys" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.499029 5515 cpu_manager.go:414] "containerMap: removing container" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.499038 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.499064 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.499072 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.499328 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.499349 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.499361 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.499375 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerName="cleanup" Oct 30 13:45:20.503101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.499393 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e53cfa-6c99-4543-84ae-be9dbadde34e" containerName="revoke-keys" Oct 30 13:45:20.507792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.507716 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.508806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.508787 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5"] Oct 30 13:45:20.520414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.520390 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:20.524088 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.524029 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:45:20.530036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.529249 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:20.552170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.552146 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:45:20.565556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.559257 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.565556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.559344 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.565556 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.559405 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d944b\" (UniqueName: \"kubernetes.io/projected/47e0ebcf-081f-4027-b015-e48962364b71-kube-api-access-d944b\") pod \"maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.577592 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.577561 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:45:20.661550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.660781 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78818086-4780-4d9b-be87-19b549bcdac2-kube-api-access-pmpzv\" (UniqueName: \"kubernetes.io/projected/78818086-4780-4d9b-be87-19b549bcdac2-kube-api-access-pmpzv\") pod \"78818086-4780-4d9b-be87-19b549bcdac2\" (UID: \"78818086-4780-4d9b-be87-19b549bcdac2\") " Oct 30 13:45:20.661550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.660911 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78818086-4780-4d9b-be87-19b549bcdac2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78818086-4780-4d9b-be87-19b549bcdac2-maas-api-service-ca\") pod \"78818086-4780-4d9b-be87-19b549bcdac2\" (UID: \"78818086-4780-4d9b-be87-19b549bcdac2\") " Oct 30 13:45:20.661550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.661006 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf045354-c623-4b61-bbd9-ab57faeb18a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf045354-c623-4b61-bbd9-ab57faeb18a2-plugins-config-volume\") pod \"bf045354-c623-4b61-bbd9-ab57faeb18a2\" (UID: \"bf045354-c623-4b61-bbd9-ab57faeb18a2\") " Oct 30 13:45:20.661550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.661040 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf045354-c623-4b61-bbd9-ab57faeb18a2-kube-api-access-w27fw\" (UniqueName: \"kubernetes.io/projected/bf045354-c623-4b61-bbd9-ab57faeb18a2-kube-api-access-w27fw\") pod \"bf045354-c623-4b61-bbd9-ab57faeb18a2\" (UID: \"bf045354-c623-4b61-bbd9-ab57faeb18a2\") " Oct 30 13:45:20.661550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.661074 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b49827-36bc-480e-b26e-82326bdc8b4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b49827-36bc-480e-b26e-82326bdc8b4b-maas-api-service-ca\") pod \"92b49827-36bc-480e-b26e-82326bdc8b4b\" (UID: \"92b49827-36bc-480e-b26e-82326bdc8b4b\") " Oct 30 13:45:20.661550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.661128 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b49827-36bc-480e-b26e-82326bdc8b4b-kube-api-access-q7cg7\" (UniqueName: \"kubernetes.io/projected/92b49827-36bc-480e-b26e-82326bdc8b4b-kube-api-access-q7cg7\") pod \"92b49827-36bc-480e-b26e-82326bdc8b4b\" (UID: \"92b49827-36bc-480e-b26e-82326bdc8b4b\") " Oct 30 13:45:20.661550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.661227 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-kube-api-access-hc2w2\" (UniqueName: \"kubernetes.io/projected/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-kube-api-access-hc2w2\") pod \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\" (UID: \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\") " Oct 30 13:45:20.661550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.661442 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-plugins-config-volume\") pod \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\" (UID: \"aff4dd5f-793d-4148-8be2-8cbe26c2eddf\") " Oct 30 13:45:20.662140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.661821 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d944b\" (UniqueName: \"kubernetes.io/projected/47e0ebcf-081f-4027-b015-e48962364b71-kube-api-access-d944b\") pod \"maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.662243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.662216 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.662448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.662263 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.662997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.662973 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.666150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.666122 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78818086-4780-4d9b-be87-19b549bcdac2-maas-api-service-ca" pod "78818086-4780-4d9b-be87-19b549bcdac2" (UID: "78818086-4780-4d9b-be87-19b549bcdac2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:20.666896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.666868 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf045354-c623-4b61-bbd9-ab57faeb18a2-plugins-config-volume" pod "bf045354-c623-4b61-bbd9-ab57faeb18a2" (UID: "bf045354-c623-4b61-bbd9-ab57faeb18a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:20.667886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.667843 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b49827-36bc-480e-b26e-82326bdc8b4b-maas-api-service-ca" pod "92b49827-36bc-480e-b26e-82326bdc8b4b" (UID: "92b49827-36bc-480e-b26e-82326bdc8b4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:20.668841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.668817 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-plugins-config-volume" pod "aff4dd5f-793d-4148-8be2-8cbe26c2eddf" (UID: "aff4dd5f-793d-4148-8be2-8cbe26c2eddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:20.677179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.677130 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-kube-api-access-hc2w2" pod "aff4dd5f-793d-4148-8be2-8cbe26c2eddf" (UID: "aff4dd5f-793d-4148-8be2-8cbe26c2eddf"). InnerVolumeSpecName "kube-api-access-hc2w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:20.684612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.679812 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78818086-4780-4d9b-be87-19b549bcdac2-kube-api-access-pmpzv" pod "78818086-4780-4d9b-be87-19b549bcdac2" (UID: "78818086-4780-4d9b-be87-19b549bcdac2"). InnerVolumeSpecName "kube-api-access-pmpzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:20.684612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.682777 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b49827-36bc-480e-b26e-82326bdc8b4b-kube-api-access-q7cg7" pod "92b49827-36bc-480e-b26e-82326bdc8b4b" (UID: "92b49827-36bc-480e-b26e-82326bdc8b4b"). InnerVolumeSpecName "kube-api-access-q7cg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:20.684612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.683178 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.700725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.700442 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf045354-c623-4b61-bbd9-ab57faeb18a2-kube-api-access-w27fw" pod "bf045354-c623-4b61-bbd9-ab57faeb18a2" (UID: "bf045354-c623-4b61-bbd9-ab57faeb18a2"). InnerVolumeSpecName "kube-api-access-w27fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:20.706114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.706083 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d944b\" (UniqueName: \"kubernetes.io/projected/47e0ebcf-081f-4027-b015-e48962364b71-kube-api-access-d944b\") pod \"maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.763766 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.763684 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmpzv\" (UniqueName: \"kubernetes.io/projected/78818086-4780-4d9b-be87-19b549bcdac2-kube-api-access-pmpzv\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.764207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.764190 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78818086-4780-4d9b-be87-19b549bcdac2-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.764309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.764292 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf045354-c623-4b61-bbd9-ab57faeb18a2-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.764405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.764314 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w27fw\" (UniqueName: \"kubernetes.io/projected/bf045354-c623-4b61-bbd9-ab57faeb18a2-kube-api-access-w27fw\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.764405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.764329 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b49827-36bc-480e-b26e-82326bdc8b4b-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.764405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.764342 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7cg7\" (UniqueName: \"kubernetes.io/projected/92b49827-36bc-480e-b26e-82326bdc8b4b-kube-api-access-q7cg7\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.764405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.764357 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2w2\" (UniqueName: \"kubernetes.io/projected/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-kube-api-access-hc2w2\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.764405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.764372 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff4dd5f-793d-4148-8be2-8cbe26c2eddf-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:20.767372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.767348 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" Oct 30 13:45:20.772851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.767467 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t" event={"ID":"78818086-4780-4d9b-be87-19b549bcdac2","Type":"ContainerDied","Data":"8cc1684d8a6470a58a6b3afd7eedd22b8636849f306de554b59dee0d1ab96806"} Oct 30 13:45:20.774317 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.774058 5515 generic.go:356] "Generic (PLEG): container finished" podID="bf045354-c623-4b61-bbd9-ab57faeb18a2" containerID="b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13" exitCode=0 Oct 30 13:45:20.774469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.774448 5515 scope.go:122] "RemoveContainer" containerID="dc79573d074002af5351b0e59458b9c06475ffb93cfb7c07a4b08577bd9d07be" Oct 30 13:45:20.774600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.774122 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" Oct 30 13:45:20.777257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.774297 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" event={"ID":"bf045354-c623-4b61-bbd9-ab57faeb18a2","Type":"ContainerDied","Data":"b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13"} Oct 30 13:45:20.777463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.777353 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf" event={"ID":"bf045354-c623-4b61-bbd9-ab57faeb18a2","Type":"ContainerDied","Data":"e86207952643ad1ad8707af2acdd05936d139578726b1a63846a205638d9697a"} Oct 30 13:45:20.784333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.782243 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" Oct 30 13:45:20.784333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.782487 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7" event={"ID":"92b49827-36bc-480e-b26e-82326bdc8b4b","Type":"ContainerDied","Data":"7c9ab71317c371b2cb8f1ac2223edca22c5ac6be2ccd15287f4fef4b19436918"} Oct 30 13:45:20.788551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.788525 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" event={"ID":"43855476-25c9-4939-bfc5-3701cf7f04e9","Type":"ContainerDied","Data":"c9e383b8b65adaf8a461614206d782a07c371d1dde3401370994a3e6f54c51a3"} Oct 30 13:45:20.788681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.788631 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn" Oct 30 13:45:20.809558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.809501 5515 scope.go:122] "RemoveContainer" containerID="b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13" Oct 30 13:45:20.813683 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.813652 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw"] Oct 30 13:45:20.818987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.818953 5515 generic.go:356] "Generic (PLEG): container finished" podID="aff4dd5f-793d-4148-8be2-8cbe26c2eddf" containerID="d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231" exitCode=0 Oct 30 13:45:20.819576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.819548 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" event={"ID":"aff4dd5f-793d-4148-8be2-8cbe26c2eddf","Type":"ContainerDied","Data":"d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231"} Oct 30 13:45:20.819707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.819688 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" Oct 30 13:45:20.819856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.819834 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx" event={"ID":"aff4dd5f-793d-4148-8be2-8cbe26c2eddf","Type":"ContainerDied","Data":"29da12fb3b764f4da23e9532e899faee8bba8cec266381c1179a62721b691a12"} Oct 30 13:45:20.838578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.838527 5515 generic.go:356] "Generic (PLEG): container finished" podID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerID="573070cfff0a1dad67418ee28d955c890815767a0832b62eb9a6d7df08628d66" exitCode=60 Oct 30 13:45:20.838878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.838855 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" event={"ID":"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c","Type":"ContainerDied","Data":"573070cfff0a1dad67418ee28d955c890815767a0832b62eb9a6d7df08628d66"} Oct 30 13:45:20.839508 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.839454 5515 scope.go:122] "RemoveContainer" containerID="573070cfff0a1dad67418ee28d955c890815767a0832b62eb9a6d7df08628d66" Oct 30 13:45:20.840111 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:20.840086 5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh_odh-ai-gateway-infra(95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" Oct 30 13:45:20.845758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.845420 5515 generic.go:356] "Generic (PLEG): container finished" podID="7c417014-5325-4155-838d-16a0e309477b" containerID="5ab6e68ce7164437c54ca663ad69e4cd10970fb35b6231e1e1d88c4429efa95c" exitCode=60 Oct 30 13:45:20.845758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.845476 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" event={"ID":"7c417014-5325-4155-838d-16a0e309477b","Type":"ContainerDied","Data":"5ab6e68ce7164437c54ca663ad69e4cd10970fb35b6231e1e1d88c4429efa95c"} Oct 30 13:45:20.845758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.845561 5515 scope.go:122] "RemoveContainer" containerID="b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13" Oct 30 13:45:20.846246 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:20.846218 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13\": container with ID starting with b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13 not found: ID does not exist" containerID="b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13" Oct 30 13:45:20.846378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.846258 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13"} err="failed to get container status \"b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13\": rpc error: code = NotFound desc = could not find container \"b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13\": container with ID starting with b350111c280eba768870b9517278724a6ba8ea340b4c1ac6cffb2bd2f4135c13 not found: ID does not exist" Oct 30 13:45:20.846378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.846349 5515 scope.go:122] "RemoveContainer" containerID="72097f4cb87c7062136503fe36ba1008243be33d308351cbbf3133066b2008a8" Oct 30 13:45:20.898247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.895521 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:20.910712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.910685 5515 scope.go:122] "RemoveContainer" containerID="81b153f4ed741c6a175830432023edb770893c6e8e1c7009e05b2f2ac292059b" Oct 30 13:45:20.912891 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.912869 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t"] Oct 30 13:45:20.923303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.920592 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7a8b03-29889465-rbv4t"] Oct 30 13:45:20.943762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.943704 5515 scope.go:122] "RemoveContainer" containerID="d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231" Oct 30 13:45:20.976054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.973510 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf"] Oct 30 13:45:20.981176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.981086 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9cc0e6-55b9b7cf59-qpthf"] Oct 30 13:45:20.995513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.994381 5515 scope.go:122] "RemoveContainer" containerID="d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231" Oct 30 13:45:20.995513 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:20.995427 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231\": container with ID starting with d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231 not found: ID does not exist" containerID="d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231" Oct 30 13:45:20.995513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.995466 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231"} err="failed to get container status \"d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231\": rpc error: code = NotFound desc = could not find container \"d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231\": container with ID starting with d49796708372364309cd7583799717106df3c04c31ad1d75d132d2a9f7d2c231 not found: ID does not exist" Oct 30 13:45:20.995513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:20.995494 5515 scope.go:122] "RemoveContainer" containerID="8ceda93c845abdb32d802bb791422efb51cf31a3491067fbe02f46f0261fd833" Oct 30 13:45:21.019049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.018656 5515 scope.go:122] "RemoveContainer" containerID="b7afa6a8e1fc762f42fb0aadb75da0b164fc03beb4cccb69a1d389a074f78741" Oct 30 13:45:21.058582 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.050608 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7"] Oct 30 13:45:21.058582 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.057456 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889465-6cxr7"] Oct 30 13:45:21.100253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.100224 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx"] Oct 30 13:45:21.107682 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.107594 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:21.108149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.108125 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9cc0e6-6557cfd444-9ltnx"] Oct 30 13:45:21.190125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.190087 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c417014-5325-4155-838d-16a0e309477b-kube-api-access-z2w4h\" (UniqueName: \"kubernetes.io/projected/7c417014-5325-4155-838d-16a0e309477b-kube-api-access-z2w4h\") pod \"7c417014-5325-4155-838d-16a0e309477b\" (UID: \"7c417014-5325-4155-838d-16a0e309477b\") " Oct 30 13:45:21.190290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.190218 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c417014-5325-4155-838d-16a0e309477b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c417014-5325-4155-838d-16a0e309477b-maas-api-service-ca\") pod \"7c417014-5325-4155-838d-16a0e309477b\" (UID: \"7c417014-5325-4155-838d-16a0e309477b\") " Oct 30 13:45:21.190646 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.190618 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c417014-5325-4155-838d-16a0e309477b-maas-api-service-ca" pod "7c417014-5325-4155-838d-16a0e309477b" (UID: "7c417014-5325-4155-838d-16a0e309477b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:21.190885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.190866 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c417014-5325-4155-838d-16a0e309477b-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:21.192368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.192342 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c417014-5325-4155-838d-16a0e309477b-kube-api-access-z2w4h" pod "7c417014-5325-4155-838d-16a0e309477b" (UID: "7c417014-5325-4155-838d-16a0e309477b"). InnerVolumeSpecName "kube-api-access-z2w4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:21.238688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.238662 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn"] Oct 30 13:45:21.254120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.254091 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-98dabb-29889465-6sxbn"] Oct 30 13:45:21.292434 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.292373 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2w4h\" (UniqueName: \"kubernetes.io/projected/7c417014-5325-4155-838d-16a0e309477b-kube-api-access-z2w4h\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:21.339599 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:45:21.339568 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e0ebcf_081f_4027_b015_e48962364b71.slice/crio-932551f17e916e5323a4d30c56ce3b42ddd7b769540ecd9a6e4b7bcfb83de5f5 WatchSource:0}: Error finding container 932551f17e916e5323a4d30c56ce3b42ddd7b769540ecd9a6e4b7bcfb83de5f5: Status 404 returned error can't find the container with id 932551f17e916e5323a4d30c56ce3b42ddd7b769540ecd9a6e4b7bcfb83de5f5 Oct 30 13:45:21.340108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.340043 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5"] Oct 30 13:45:21.863091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.862997 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" event={"ID":"7c417014-5325-4155-838d-16a0e309477b","Type":"ContainerDied","Data":"25e50a7e5616d2fe593343a9fb210f67750de831be5b83dd29da8fe0661c299a"} Oct 30 13:45:21.863091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.863069 5515 scope.go:122] "RemoveContainer" containerID="5ab6e68ce7164437c54ca663ad69e4cd10970fb35b6231e1e1d88c4429efa95c" Oct 30 13:45:21.863678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.863183 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw" Oct 30 13:45:21.878550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.876608 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" event={"ID":"47e0ebcf-081f-4027-b015-e48962364b71","Type":"ContainerStarted","Data":"5aaa5fcbfcacc66252a496d43a30d4e2703335c77f69a5516b712ee4bbff226f"} Oct 30 13:45:21.894612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.894558 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" event={"ID":"47e0ebcf-081f-4027-b015-e48962364b71","Type":"ContainerStarted","Data":"932551f17e916e5323a4d30c56ce3b42ddd7b769540ecd9a6e4b7bcfb83de5f5"} Oct 30 13:45:21.945083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.945028 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" podStartSLOduration=1.945014849 podStartE2EDuration="1.945014849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:21.943583249 +0000 UTC m=+4340.070384148" watchObservedRunningTime="2026-10-30 13:45:21.945014849 +0000 UTC m=+4340.071815800" Oct 30 13:45:21.992254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:21.992221 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw"] Oct 30 13:45:22.015949 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.015916 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" path="/var/lib/kubelet/pods/43855476-25c9-4939-bfc5-3701cf7f04e9/volumes" Oct 30 13:45:22.016658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.016635 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78818086-4780-4d9b-be87-19b549bcdac2" path="/var/lib/kubelet/pods/78818086-4780-4d9b-be87-19b549bcdac2/volumes" Oct 30 13:45:22.017108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.017085 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" path="/var/lib/kubelet/pods/92b49827-36bc-480e-b26e-82326bdc8b4b/volumes" Oct 30 13:45:22.017920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.017900 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff4dd5f-793d-4148-8be2-8cbe26c2eddf" path="/var/lib/kubelet/pods/aff4dd5f-793d-4148-8be2-8cbe26c2eddf/volumes" Oct 30 13:45:22.018330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.018316 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf045354-c623-4b61-bbd9-ab57faeb18a2" path="/var/lib/kubelet/pods/bf045354-c623-4b61-bbd9-ab57faeb18a2/volumes" Oct 30 13:45:22.018709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.018692 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2d2a0b-29889465-96kgw"] Oct 30 13:45:22.179558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.179527 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh"] Oct 30 13:45:22.287907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.287872 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:22.394414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.394388 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:22.518016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.517984 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-kube-api-access-zc6l7\" (UniqueName: \"kubernetes.io/projected/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-kube-api-access-zc6l7\") pod \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\" (UID: \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\") " Oct 30 13:45:22.518146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.518090 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-maas-api-service-ca\") pod \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\" (UID: \"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c\") " Oct 30 13:45:22.518478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.518449 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-maas-api-service-ca" pod "95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" (UID: "95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:22.518769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.518745 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-maas-api-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:22.520165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.520146 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-kube-api-access-zc6l7" pod "95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" (UID: "95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c"). InnerVolumeSpecName "kube-api-access-zc6l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:22.620750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.620721 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc6l7\" (UniqueName: \"kubernetes.io/projected/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c-kube-api-access-zc6l7\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:22.886777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.886740 5515 generic.go:356] "Generic (PLEG): container finished" podID="47e0ebcf-081f-4027-b015-e48962364b71" containerID="5aaa5fcbfcacc66252a496d43a30d4e2703335c77f69a5516b712ee4bbff226f" exitCode=0 Oct 30 13:45:22.887252 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.886821 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" event={"ID":"47e0ebcf-081f-4027-b015-e48962364b71","Type":"ContainerDied","Data":"5aaa5fcbfcacc66252a496d43a30d4e2703335c77f69a5516b712ee4bbff226f"} Oct 30 13:45:22.888400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.888373 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" event={"ID":"95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c","Type":"ContainerDied","Data":"e8b52fa8ce4a91507325029ae1a8c4bde38304561293c0b9e93f9b40642cfd34"} Oct 30 13:45:22.888530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.888432 5515 scope.go:122] "RemoveContainer" containerID="573070cfff0a1dad67418ee28d955c890815767a0832b62eb9a6d7df08628d66" Oct 30 13:45:22.888530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.888383 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh" Oct 30 13:45:22.976004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.975973 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh"] Oct 30 13:45:22.977016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:22.976816 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889465-npvmh"] Oct 30 13:45:24.059438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.057220 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:24.080647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.080596 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c417014-5325-4155-838d-16a0e309477b" path="/var/lib/kubelet/pods/7c417014-5325-4155-838d-16a0e309477b/volumes" Oct 30 13:45:24.081286 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.081237 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" path="/var/lib/kubelet/pods/95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c/volumes" Oct 30 13:45:24.138339 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.138312 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:24.257251 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.257163 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e0ebcf-081f-4027-b015-e48962364b71-kube-api-access-d944b\" (UniqueName: \"kubernetes.io/projected/47e0ebcf-081f-4027-b015-e48962364b71-kube-api-access-d944b\") pod \"47e0ebcf-081f-4027-b015-e48962364b71\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " Oct 30 13:45:24.257525 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.257508 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-openshift-ca\") pod \"47e0ebcf-081f-4027-b015-e48962364b71\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " Oct 30 13:45:24.257586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.257574 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-xks-ca\") pod \"47e0ebcf-081f-4027-b015-e48962364b71\" (UID: \"47e0ebcf-081f-4027-b015-e48962364b71\") " Oct 30 13:45:24.257893 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.257865 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-openshift-ca" pod "47e0ebcf-081f-4027-b015-e48962364b71" (UID: "47e0ebcf-081f-4027-b015-e48962364b71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:24.258242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.258220 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-openshift-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.266492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.266460 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-xks-ca" pod "47e0ebcf-081f-4027-b015-e48962364b71" (UID: "47e0ebcf-081f-4027-b015-e48962364b71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:24.266492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.266483 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e0ebcf-081f-4027-b015-e48962364b71-kube-api-access-d944b" pod "47e0ebcf-081f-4027-b015-e48962364b71" (UID: "47e0ebcf-081f-4027-b015-e48962364b71"). InnerVolumeSpecName "kube-api-access-d944b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:24.360036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.360002 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e0ebcf-081f-4027-b015-e48962364b71-maas-api-xks-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.360036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.360031 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d944b\" (UniqueName: \"kubernetes.io/projected/47e0ebcf-081f-4027-b015-e48962364b71-kube-api-access-d944b\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:24.567449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.567341 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6"] Oct 30 13:45:24.710289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.708578 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:24.928481 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.928453 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" Oct 30 13:45:24.928675 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.928453 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5" event={"ID":"47e0ebcf-081f-4027-b015-e48962364b71","Type":"ContainerDied","Data":"932551f17e916e5323a4d30c56ce3b42ddd7b769540ecd9a6e4b7bcfb83de5f5"} Oct 30 13:45:24.928675 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.928538 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="932551f17e916e5323a4d30c56ce3b42ddd7b769540ecd9a6e4b7bcfb83de5f5" Oct 30 13:45:24.935890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:24.935861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:25.284343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:25.281399 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:26.070628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:26.070596 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:27.906647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:27.906528 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:28.993029 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:28.992998 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:45:29.960404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:29.960361 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:31.191294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:31.190217 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:31.407865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:31.407826 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:31.623337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:31.620959 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:33.282839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:33.282801 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:34.997657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:34.997622 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:35.556960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:35.556919 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:35.761183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:35.761145 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:36.126145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:36.126113 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq"] Oct 30 13:45:36.407345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:36.407306 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:36.620531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:36.620488 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:36.839328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:36.839214 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:38.069982 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.069948 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv"] Oct 30 13:45:38.070494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.070188 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" podUID="8130ae3f-6e4d-428d-a10c-b830ee51e6da" containerName="maas-api" containerID="cri-o://213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142" gracePeriod=30 Oct 30 13:45:38.223312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.223284 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:38.493664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.493632 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:45:38.571097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.571064 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8130ae3f-6e4d-428d-a10c-b830ee51e6da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8130ae3f-6e4d-428d-a10c-b830ee51e6da-openshift-service-ca\") pod \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " Oct 30 13:45:38.571546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.571511 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-maas-api-tls\") pod \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " Oct 30 13:45:38.571674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.571639 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-metrics-certs\") pod \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " Oct 30 13:45:38.571787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.571771 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8130ae3f-6e4d-428d-a10c-b830ee51e6da-kube-api-access-qb9ck\" (UniqueName: \"kubernetes.io/projected/8130ae3f-6e4d-428d-a10c-b830ee51e6da-kube-api-access-qb9ck\") pod \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\" (UID: \"8130ae3f-6e4d-428d-a10c-b830ee51e6da\") " Oct 30 13:45:38.572808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.572783 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8130ae3f-6e4d-428d-a10c-b830ee51e6da-openshift-service-ca" pod "8130ae3f-6e4d-428d-a10c-b830ee51e6da" (UID: "8130ae3f-6e4d-428d-a10c-b830ee51e6da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:38.575998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.575726 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-metrics-certs" pod "8130ae3f-6e4d-428d-a10c-b830ee51e6da" (UID: "8130ae3f-6e4d-428d-a10c-b830ee51e6da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:38.577006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.576977 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-maas-api-tls" pod "8130ae3f-6e4d-428d-a10c-b830ee51e6da" (UID: "8130ae3f-6e4d-428d-a10c-b830ee51e6da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:38.577738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.577644 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8130ae3f-6e4d-428d-a10c-b830ee51e6da-kube-api-access-qb9ck" pod "8130ae3f-6e4d-428d-a10c-b830ee51e6da" (UID: "8130ae3f-6e4d-428d-a10c-b830ee51e6da"). InnerVolumeSpecName "kube-api-access-qb9ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:38.673511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.673431 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8130ae3f-6e4d-428d-a10c-b830ee51e6da-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:38.673511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.673459 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:38.673511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.673469 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8130ae3f-6e4d-428d-a10c-b830ee51e6da-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:38.673511 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.673477 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb9ck\" (UniqueName: \"kubernetes.io/projected/8130ae3f-6e4d-428d-a10c-b830ee51e6da-kube-api-access-qb9ck\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:38.706127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.706043 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:38.920815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:38.920769 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:39.110359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.110229 5515 generic.go:356] "Generic (PLEG): container finished" podID="8130ae3f-6e4d-428d-a10c-b830ee51e6da" containerID="213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142" exitCode=0 Oct 30 13:45:39.110359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.110295 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" Oct 30 13:45:39.110359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.110304 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" event={"ID":"8130ae3f-6e4d-428d-a10c-b830ee51e6da","Type":"ContainerDied","Data":"213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142"} Oct 30 13:45:39.110359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.110355 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv" event={"ID":"8130ae3f-6e4d-428d-a10c-b830ee51e6da","Type":"ContainerDied","Data":"8e661abd8d122fa1ab05f8b389307487dedf4f1c50c509cbfc75a52bbe6a92eb"} Oct 30 13:45:39.111021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.110385 5515 scope.go:122] "RemoveContainer" containerID="213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142" Oct 30 13:45:39.123111 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.123090 5515 scope.go:122] "RemoveContainer" containerID="213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142" Oct 30 13:45:39.123453 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:39.123429 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142\": container with ID starting with 213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142 not found: ID does not exist" containerID="213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142" Oct 30 13:45:39.123521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.123462 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142"} err="failed to get container status \"213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142\": rpc error: code = NotFound desc = could not find container \"213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142\": container with ID starting with 213271f66b056a832801d7f06c8610c6001bce23b288cbc52edd07f5db244142 not found: ID does not exist" Oct 30 13:45:39.180509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.180465 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv"] Oct 30 13:45:39.180701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.180531 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9cc0e6-7479bf6fcc-dkqjv"] Oct 30 13:45:39.476984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.476953 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:39.590129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.590086 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" podUID="cc50345e-fefa-4329-a72c-2248c950a603" containerName="main" containerID="cri-o://ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd" gracePeriod=45 Oct 30 13:45:39.675146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.675100 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" podUID="cc50345e-fefa-4329-a72c-2248c950a603" containerName="main" probeResult="failure" output="Get \"https://10.134.1.1:8000/ready\": dial tcp 10.134.1.1:8000: connect: connection refused" Oct 30 13:45:39.895812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.895789 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:45:39.996712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.996676 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-model-cache\") pod \"cc50345e-fefa-4329-a72c-2248c950a603\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " Oct 30 13:45:39.996933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.996814 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-dshm\") pod \"cc50345e-fefa-4329-a72c-2248c950a603\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " Oct 30 13:45:39.996933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.996915 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc50345e-fefa-4329-a72c-2248c950a603-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc50345e-fefa-4329-a72c-2248c950a603-tls-certs\") pod \"cc50345e-fefa-4329-a72c-2248c950a603\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " Oct 30 13:45:39.997046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.996976 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-model-cache" pod "cc50345e-fefa-4329-a72c-2248c950a603" (UID: "cc50345e-fefa-4329-a72c-2248c950a603"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:39.997095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.997064 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-tmp-dir\") pod \"cc50345e-fefa-4329-a72c-2248c950a603\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " Oct 30 13:45:39.997749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.997196 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc50345e-fefa-4329-a72c-2248c950a603-kube-api-access-sh9zx\" (UniqueName: \"kubernetes.io/projected/cc50345e-fefa-4329-a72c-2248c950a603-kube-api-access-sh9zx\") pod \"cc50345e-fefa-4329-a72c-2248c950a603\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " Oct 30 13:45:39.997749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.997254 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-home\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-home\") pod \"cc50345e-fefa-4329-a72c-2248c950a603\" (UID: \"cc50345e-fefa-4329-a72c-2248c950a603\") " Oct 30 13:45:39.997749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.997503 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-tmp-dir" pod "cc50345e-fefa-4329-a72c-2248c950a603" (UID: "cc50345e-fefa-4329-a72c-2248c950a603"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:39.997749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.997724 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-home" pod "cc50345e-fefa-4329-a72c-2248c950a603" (UID: "cc50345e-fefa-4329-a72c-2248c950a603"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:39.998166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.998141 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:39.998166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.998171 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:39.998329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.998187 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:40.000769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.999353 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-dshm" pod "cc50345e-fefa-4329-a72c-2248c950a603" (UID: "cc50345e-fefa-4329-a72c-2248c950a603"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:40.000769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:39.999474 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc50345e-fefa-4329-a72c-2248c950a603-kube-api-access-sh9zx" pod "cc50345e-fefa-4329-a72c-2248c950a603" (UID: "cc50345e-fefa-4329-a72c-2248c950a603"). InnerVolumeSpecName "kube-api-access-sh9zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:40.001201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.001174 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc50345e-fefa-4329-a72c-2248c950a603-tls-certs" pod "cc50345e-fefa-4329-a72c-2248c950a603" (UID: "cc50345e-fefa-4329-a72c-2248c950a603"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:40.006100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.006075 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8130ae3f-6e4d-428d-a10c-b830ee51e6da" path="/var/lib/kubelet/pods/8130ae3f-6e4d-428d-a10c-b830ee51e6da/volumes" Oct 30 13:45:40.100477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.100445 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc50345e-fefa-4329-a72c-2248c950a603-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:40.100477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.100475 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh9zx\" (UniqueName: \"kubernetes.io/projected/cc50345e-fefa-4329-a72c-2248c950a603-kube-api-access-sh9zx\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:40.100477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.100486 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc50345e-fefa-4329-a72c-2248c950a603-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:40.121410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.121379 5515 generic.go:356] "Generic (PLEG): container finished" podID="cc50345e-fefa-4329-a72c-2248c950a603" containerID="ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd" exitCode=0 Oct 30 13:45:40.121847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.121418 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" Oct 30 13:45:40.121847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.121442 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" event={"ID":"cc50345e-fefa-4329-a72c-2248c950a603","Type":"ContainerDied","Data":"ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd"} Oct 30 13:45:40.121847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.121482 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6" event={"ID":"cc50345e-fefa-4329-a72c-2248c950a603","Type":"ContainerDied","Data":"efa64e8766964b829183c88bb23f53d534f41014ffee2f46e3f77b9e3cfdaa25"} Oct 30 13:45:40.121847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.121500 5515 scope.go:122] "RemoveContainer" containerID="ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd" Oct 30 13:45:40.132606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.132587 5515 scope.go:122] "RemoveContainer" containerID="ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd" Oct 30 13:45:40.132897 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:40.132878 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd\": container with ID starting with ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd not found: ID does not exist" containerID="ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd" Oct 30 13:45:40.132948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.132906 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd"} err="failed to get container status \"ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd\": rpc error: code = NotFound desc = could not find container \"ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd\": container with ID starting with ced2ae2b99ca32773b41bc64387e273b4febed789e29e72a7c0d98060bae0efd not found: ID does not exist" Oct 30 13:45:40.175330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.175296 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6"] Oct 30 13:45:40.180314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.180283 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-af8369/test-model-af8369-kserve-6d85cc47b5-lpvg6"] Oct 30 13:45:40.522140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:40.522104 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:41.116660 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.116627 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:41.127151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.127112 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" podUID="27c5eebc-f9da-442c-81b9-831d45fe4c01" containerName="payload-processing" containerID="cri-o://7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad" gracePeriod=25 Oct 30 13:45:41.219309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.217139 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5"] Oct 30 13:45:41.234637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.234592 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-025c7f-t95p5"] Oct 30 13:45:41.385135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.385101 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:41.415152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.415128 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:45:41.542283 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.542248 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c5eebc-f9da-442c-81b9-831d45fe4c01-kube-api-access-4jbqw\" (UniqueName: \"kubernetes.io/projected/27c5eebc-f9da-442c-81b9-831d45fe4c01-kube-api-access-4jbqw\") pod \"27c5eebc-f9da-442c-81b9-831d45fe4c01\" (UID: \"27c5eebc-f9da-442c-81b9-831d45fe4c01\") " Oct 30 13:45:41.542497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.542370 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c5eebc-f9da-442c-81b9-831d45fe4c01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c5eebc-f9da-442c-81b9-831d45fe4c01-plugins-config-volume\") pod \"27c5eebc-f9da-442c-81b9-831d45fe4c01\" (UID: \"27c5eebc-f9da-442c-81b9-831d45fe4c01\") " Oct 30 13:45:41.542793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.542770 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c5eebc-f9da-442c-81b9-831d45fe4c01-plugins-config-volume" pod "27c5eebc-f9da-442c-81b9-831d45fe4c01" (UID: "27c5eebc-f9da-442c-81b9-831d45fe4c01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:41.543138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.543116 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c5eebc-f9da-442c-81b9-831d45fe4c01-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:41.544537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.544514 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c5eebc-f9da-442c-81b9-831d45fe4c01-kube-api-access-4jbqw" pod "27c5eebc-f9da-442c-81b9-831d45fe4c01" (UID: "27c5eebc-f9da-442c-81b9-831d45fe4c01"). InnerVolumeSpecName "kube-api-access-4jbqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:41.644807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.644775 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbqw\" (UniqueName: \"kubernetes.io/projected/27c5eebc-f9da-442c-81b9-831d45fe4c01-kube-api-access-4jbqw\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:41.885095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:41.885055 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:42.003678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.003590 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e0ebcf-081f-4027-b015-e48962364b71" path="/var/lib/kubelet/pods/47e0ebcf-081f-4027-b015-e48962364b71/volumes" Oct 30 13:45:42.004210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.004186 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc50345e-fefa-4329-a72c-2248c950a603" path="/var/lib/kubelet/pods/cc50345e-fefa-4329-a72c-2248c950a603/volumes" Oct 30 13:45:42.140156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.140122 5515 generic.go:356] "Generic (PLEG): container finished" podID="27c5eebc-f9da-442c-81b9-831d45fe4c01" containerID="7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad" exitCode=0 Oct 30 13:45:42.141074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.140163 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" Oct 30 13:45:42.141074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.140201 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" event={"ID":"27c5eebc-f9da-442c-81b9-831d45fe4c01","Type":"ContainerDied","Data":"7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad"} Oct 30 13:45:42.141074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.140256 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq" event={"ID":"27c5eebc-f9da-442c-81b9-831d45fe4c01","Type":"ContainerDied","Data":"e04c03d1fe37d5e5f4446a5f7dc5dbb046997df7346df538a325c8b54dc0db9b"} Oct 30 13:45:42.141074 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.140314 5515 scope.go:122] "RemoveContainer" containerID="7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad" Oct 30 13:45:42.152121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.152103 5515 scope.go:122] "RemoveContainer" containerID="7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad" Oct 30 13:45:42.152456 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:42.152432 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad\": container with ID starting with 7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad not found: ID does not exist" containerID="7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad" Oct 30 13:45:42.152538 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.152464 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad"} err="failed to get container status \"7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad\": rpc error: code = NotFound desc = could not find container \"7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad\": container with ID starting with 7ec804afaa30c2bfe0777961fb75e9be1cbb2fadd72f50b00daf0adb1cc19fad not found: ID does not exist" Oct 30 13:45:42.199634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.199609 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq"] Oct 30 13:45:42.211960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:42.211932 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b7407-8575d85884-72pjq"] Oct 30 13:45:43.079302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:43.079252 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:44.019065 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:44.019022 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c5eebc-f9da-442c-81b9-831d45fe4c01" path="/var/lib/kubelet/pods/27c5eebc-f9da-442c-81b9-831d45fe4c01/volumes" Oct 30 13:45:44.778293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:44.775190 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:46.137596 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:46.137566 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:46.413203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:46.413108 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:46.622011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:46.621976 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:48.438189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:48.438160 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:48.941565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:48.941538 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:49.176887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:49.176836 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:49.220476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:49.220397 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b"] Oct 30 13:45:49.220654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:49.220633 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" containerID="cri-o://d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28" gracePeriod=30 Oct 30 13:45:49.464733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:49.464701 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:50.473755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:50.473724 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:51.982061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:51.982024 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8"] Oct 30 13:45:51.982929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:51.982896 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" podUID="b36e1b90-f130-4b8d-be62-197d45ef6bc5" containerName="maas-api" containerID="cri-o://660f1ccaec9e77437b7b806157894648ccb46b751c357e8f8114a17e3f0f850b" gracePeriod=30 Oct 30 13:45:51.993353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:51.992931 5515 patch_prober.go:29] interesting pod/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:45:51.993353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:51.992975 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:45:52.164641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.164606 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:52.228558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.228530 5515 generic.go:356] "Generic (PLEG): container finished" podID="b36e1b90-f130-4b8d-be62-197d45ef6bc5" containerID="660f1ccaec9e77437b7b806157894648ccb46b751c357e8f8114a17e3f0f850b" exitCode=0 Oct 30 13:45:52.228733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.228606 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" event={"ID":"b36e1b90-f130-4b8d-be62-197d45ef6bc5","Type":"ContainerDied","Data":"660f1ccaec9e77437b7b806157894648ccb46b751c357e8f8114a17e3f0f850b"} Oct 30 13:45:52.347475 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.347441 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:45:52.417226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.417184 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls\") pod \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " Oct 30 13:45:52.417683 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.417656 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-metrics-certs\") pod \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " Oct 30 13:45:52.417892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.417868 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36e1b90-f130-4b8d-be62-197d45ef6bc5-kube-api-access-4xs4x\" (UniqueName: \"kubernetes.io/projected/b36e1b90-f130-4b8d-be62-197d45ef6bc5-kube-api-access-4xs4x\") pod \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " Oct 30 13:45:52.418076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.417956 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36e1b90-f130-4b8d-be62-197d45ef6bc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36e1b90-f130-4b8d-be62-197d45ef6bc5-openshift-service-ca\") pod \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\" (UID: \"b36e1b90-f130-4b8d-be62-197d45ef6bc5\") " Oct 30 13:45:52.418345 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.418318 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36e1b90-f130-4b8d-be62-197d45ef6bc5-openshift-service-ca" pod "b36e1b90-f130-4b8d-be62-197d45ef6bc5" (UID: "b36e1b90-f130-4b8d-be62-197d45ef6bc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:52.418941 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.418916 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36e1b90-f130-4b8d-be62-197d45ef6bc5-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:52.419517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.419494 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls" pod "b36e1b90-f130-4b8d-be62-197d45ef6bc5" (UID: "b36e1b90-f130-4b8d-be62-197d45ef6bc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:52.420656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.420630 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-metrics-certs" pod "b36e1b90-f130-4b8d-be62-197d45ef6bc5" (UID: "b36e1b90-f130-4b8d-be62-197d45ef6bc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:52.422426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.422400 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36e1b90-f130-4b8d-be62-197d45ef6bc5-kube-api-access-4xs4x" pod "b36e1b90-f130-4b8d-be62-197d45ef6bc5" (UID: "b36e1b90-f130-4b8d-be62-197d45ef6bc5"). InnerVolumeSpecName "kube-api-access-4xs4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:52.537376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.521077 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:52.537376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.521112 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36e1b90-f130-4b8d-be62-197d45ef6bc5-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:52.537376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.521125 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xs4x\" (UniqueName: \"kubernetes.io/projected/b36e1b90-f130-4b8d-be62-197d45ef6bc5-kube-api-access-4xs4x\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:52.972469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:52.972435 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:53.182600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:53.182567 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:53.237944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:53.237863 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" Oct 30 13:45:53.237944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:53.237897 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8" event={"ID":"b36e1b90-f130-4b8d-be62-197d45ef6bc5","Type":"ContainerDied","Data":"413ee5333f80ba8b65d8040d99775425fa7b04820d39834f2743c917d2e2feba"} Oct 30 13:45:53.238162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:53.237965 5515 scope.go:122] "RemoveContainer" containerID="660f1ccaec9e77437b7b806157894648ccb46b751c357e8f8114a17e3f0f850b" Oct 30 13:45:53.294216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:53.294186 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8"] Oct 30 13:45:53.307641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:53.307616 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09ad1e-94598c5bf-r5ln8"] Oct 30 13:45:53.603547 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:53.603478 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:54.011089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.011048 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36e1b90-f130-4b8d-be62-197d45ef6bc5" path="/var/lib/kubelet/pods/b36e1b90-f130-4b8d-be62-197d45ef6bc5/volumes" Oct 30 13:45:54.542891 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.542868 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:45:54.656288 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656233 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-credential-socket\") pod \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " Oct 30 13:45:54.656503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656339 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-token\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-token\") pod \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " Oct 30 13:45:54.656503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656412 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-data\") pod \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " Oct 30 13:45:54.656503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656491 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-kube-api-access-9xctw\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-kube-api-access-9xctw\") pod \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " Oct 30 13:45:54.656695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656542 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istiod-ca-cert\") pod \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " Oct 30 13:45:54.656695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656555 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-credential-socket" pod "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" (UID: "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:54.656695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656588 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-envoy\") pod \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " Oct 30 13:45:54.656873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656699 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-socket\") pod \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " Oct 30 13:45:54.656873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656732 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-certs\") pod \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " Oct 30 13:45:54.656873 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.656786 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-podinfo\") pod \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\" (UID: \"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3\") " Oct 30 13:45:54.657943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.657521 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:54.657943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.657713 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-certs" pod "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" (UID: "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:54.657943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.657814 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istiod-ca-cert" pod "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" (UID: "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:54.657943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.657853 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-socket" pod "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" (UID: "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:54.657943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.657910 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-data" pod "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" (UID: "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:54.660243 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.660217 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-envoy" pod "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" (UID: "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:54.660587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.660535 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-kube-api-access-9xctw" pod "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" (UID: "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3"). InnerVolumeSpecName "kube-api-access-9xctw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:54.660587 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.660535 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-podinfo" pod "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" (UID: "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:54.660749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.660677 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-token" pod "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" (UID: "38a6b23d-d1b5-45f1-bbca-3185a13f5bf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:54.758648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.758609 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:54.758648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.758639 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:54.758648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.758653 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:54.758894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.758665 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xctw\" (UniqueName: \"kubernetes.io/projected/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-kube-api-access-9xctw\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:54.758894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.758679 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:54.758894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.758691 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:54.758894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.758702 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:54.758894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:54.758713 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:45:55.148956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.148923 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:55.260041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.260002 5515 generic.go:356] "Generic (PLEG): container finished" podID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerID="d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28" exitCode=0 Oct 30 13:45:55.260242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.260055 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" Oct 30 13:45:55.260242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.260109 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" event={"ID":"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3","Type":"ContainerDied","Data":"d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28"} Oct 30 13:45:55.260242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.260169 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b" event={"ID":"38a6b23d-d1b5-45f1-bbca-3185a13f5bf3","Type":"ContainerDied","Data":"ef75176488c74bb55e6103d93bb865c87f8643169711386a1d4bb234b47bc680"} Oct 30 13:45:55.260242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.260195 5515 scope.go:122] "RemoveContainer" containerID="d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28" Oct 30 13:45:55.276676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.276571 5515 scope.go:122] "RemoveContainer" containerID="d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28" Oct 30 13:45:55.277393 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:55.277334 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28\": container with ID starting with d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28 not found: ID does not exist" containerID="d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28" Oct 30 13:45:55.277393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.277373 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28"} err="failed to get container status \"d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28\": rpc error: code = NotFound desc = could not find container \"d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28\": container with ID starting with d00e1b403e428b526dbbbfa22f373c1a94573e2ac44bf02454f1f8ae337c8c28 not found: ID does not exist" Oct 30 13:45:55.338358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.338326 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b"] Oct 30 13:45:55.349896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:55.349872 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-025c7f-openshift-default-58f9fbc845-k9n4b"] Oct 30 13:45:56.005059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:56.005020 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" path="/var/lib/kubelet/pods/38a6b23d-d1b5-45f1-bbca-3185a13f5bf3/volumes" Oct 30 13:45:56.834696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:56.834631 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:57.477065 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:57.477030 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx"] Oct 30 13:45:57.731931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:57.729483 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:57.936408 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:57.936379 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:58.568901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:45:58.568871 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:45:58.989528 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:45:58.989486 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:46:00.213249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:00.213212 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:00.885631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:00.885596 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:01.109158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:01.109132 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:01.440292 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:01.431841 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:02.029948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:02.029910 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:02.477799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:02.477754 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" podUID="b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" containerName="payload-pre-processing" containerID="cri-o://c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8" gracePeriod=25 Oct 30 13:46:03.897642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:03.897604 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:05.687992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:05.687959 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:07.520926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:07.520895 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:09.428636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:09.428600 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:11.051699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:11.051667 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:11.745367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:11.745327 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" podUID="b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:46:12.615680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:12.615643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:12.860012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:12.859982 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:14.368446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:14.368413 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:16.021034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:16.020989 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:16.227334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:16.227298 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:17.715342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:17.715305 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:18.942069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:18.942036 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j"] Oct 30 13:46:18.942633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:18.942293 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" podUID="c460355f-0748-412a-a6f6-eb0a0548c441" containerName="maas-api" containerID="cri-o://14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642" gracePeriod=30 Oct 30 13:46:19.096204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.096170 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:19.316500 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.316468 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:46:19.442198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.442146 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c460355f-0748-412a-a6f6-eb0a0548c441-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c460355f-0748-412a-a6f6-eb0a0548c441-openshift-service-ca\") pod \"c460355f-0748-412a-a6f6-eb0a0548c441\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " Oct 30 13:46:19.442427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.442405 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c460355f-0748-412a-a6f6-eb0a0548c441-kube-api-access-72tgx\" (UniqueName: \"kubernetes.io/projected/c460355f-0748-412a-a6f6-eb0a0548c441-kube-api-access-72tgx\") pod \"c460355f-0748-412a-a6f6-eb0a0548c441\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " Oct 30 13:46:19.442649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.442629 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls\") pod \"c460355f-0748-412a-a6f6-eb0a0548c441\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " Oct 30 13:46:19.442713 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.442655 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c460355f-0748-412a-a6f6-eb0a0548c441-openshift-service-ca" pod "c460355f-0748-412a-a6f6-eb0a0548c441" (UID: "c460355f-0748-412a-a6f6-eb0a0548c441"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:46:19.442779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.442738 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-metrics-certs\") pod \"c460355f-0748-412a-a6f6-eb0a0548c441\" (UID: \"c460355f-0748-412a-a6f6-eb0a0548c441\") " Oct 30 13:46:19.443422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.443394 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c460355f-0748-412a-a6f6-eb0a0548c441-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:19.445407 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.445374 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls" pod "c460355f-0748-412a-a6f6-eb0a0548c441" (UID: "c460355f-0748-412a-a6f6-eb0a0548c441"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:46:19.445527 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.445473 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c460355f-0748-412a-a6f6-eb0a0548c441-kube-api-access-72tgx" pod "c460355f-0748-412a-a6f6-eb0a0548c441" (UID: "c460355f-0748-412a-a6f6-eb0a0548c441"). InnerVolumeSpecName "kube-api-access-72tgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:46:19.445593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.445557 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-metrics-certs" pod "c460355f-0748-412a-a6f6-eb0a0548c441" (UID: "c460355f-0748-412a-a6f6-eb0a0548c441"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:46:19.462503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.462469 5515 generic.go:356] "Generic (PLEG): container finished" podID="c460355f-0748-412a-a6f6-eb0a0548c441" containerID="14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642" exitCode=0 Oct 30 13:46:19.462678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.462521 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" Oct 30 13:46:19.462678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.462551 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" event={"ID":"c460355f-0748-412a-a6f6-eb0a0548c441","Type":"ContainerDied","Data":"14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642"} Oct 30 13:46:19.462678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.462605 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j" event={"ID":"c460355f-0748-412a-a6f6-eb0a0548c441","Type":"ContainerDied","Data":"1d3f07dc28a9b7db7bb3664809dc8f0dc7d73098a0216462e938f9cdaa8ca37a"} Oct 30 13:46:19.462678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.462626 5515 scope.go:122] "RemoveContainer" containerID="14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642" Oct 30 13:46:19.481777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.481738 5515 scope.go:122] "RemoveContainer" containerID="14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642" Oct 30 13:46:19.482152 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:46:19.482123 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642\": container with ID starting with 14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642 not found: ID does not exist" containerID="14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642" Oct 30 13:46:19.482326 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.482165 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642"} err="failed to get container status \"14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642\": rpc error: code = NotFound desc = could not find container \"14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642\": container with ID starting with 14dd91c0837866ec3d3443bb258bfb8373d5084d5d28418590841649b2c6b642 not found: ID does not exist" Oct 30 13:46:19.534521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.534428 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j"] Oct 30 13:46:19.544348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.544309 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7b7407-56664dd49c-vwq2j"] Oct 30 13:46:19.545127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.545093 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:19.545127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.545128 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c460355f-0748-412a-a6f6-eb0a0548c441-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:19.545342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.545141 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72tgx\" (UniqueName: \"kubernetes.io/projected/c460355f-0748-412a-a6f6-eb0a0548c441-kube-api-access-72tgx\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:19.667170 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.667137 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:19.879326 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:19.879192 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:20.004495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:20.004462 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c460355f-0748-412a-a6f6-eb0a0548c441" path="/var/lib/kubelet/pods/c460355f-0748-412a-a6f6-eb0a0548c441/volumes" Oct 30 13:46:20.372762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:20.372733 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:20.637929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:20.637898 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:21.615505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:21.615474 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:21.745238 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:21.745196 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" podUID="b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:46:22.776684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:22.776657 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:46:22.799964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:22.799927 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-plugins-config-volume\") pod \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\" (UID: \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\") " Oct 30 13:46:22.800159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:22.799980 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-kube-api-access-42hsd\" (UniqueName: \"kubernetes.io/projected/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-kube-api-access-42hsd\") pod \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\" (UID: \"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b\") " Oct 30 13:46:22.800432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:22.800404 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-plugins-config-volume" pod "b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" (UID: "b5a8fa96-a0d6-4a71-b7cf-75c02aab471b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:46:22.800694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:22.800671 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:22.802481 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:22.802455 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-kube-api-access-42hsd" pod "b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" (UID: "b5a8fa96-a0d6-4a71-b7cf-75c02aab471b"). InnerVolumeSpecName "kube-api-access-42hsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:46:22.902160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:22.902123 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42hsd\" (UniqueName: \"kubernetes.io/projected/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b-kube-api-access-42hsd\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:23.398796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.398762 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:23.497433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.497391 5515 generic.go:356] "Generic (PLEG): container finished" podID="b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" containerID="c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8" exitCode=0 Oct 30 13:46:23.497642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.497436 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" Oct 30 13:46:23.497642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.497471 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" event={"ID":"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b","Type":"ContainerDied","Data":"c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8"} Oct 30 13:46:23.497642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.497529 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx" event={"ID":"b5a8fa96-a0d6-4a71-b7cf-75c02aab471b","Type":"ContainerDied","Data":"05ca6806726a7f4c70f714721263d947913ce035e5b67edb0a0ca0d97c6ed36f"} Oct 30 13:46:23.497642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.497556 5515 scope.go:122] "RemoveContainer" containerID="c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8" Oct 30 13:46:23.509535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.509514 5515 scope.go:122] "RemoveContainer" containerID="c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8" Oct 30 13:46:23.509867 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:46:23.509834 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8\": container with ID starting with c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8 not found: ID does not exist" containerID="c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8" Oct 30 13:46:23.509995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.509869 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8"} err="failed to get container status \"c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8\": rpc error: code = NotFound desc = could not find container \"c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8\": container with ID starting with c244726a7cba55cd062184c1582c91b51e0f19630fbf9e47d877e2c0b976d7c8 not found: ID does not exist" Oct 30 13:46:23.557757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.557716 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx"] Oct 30 13:46:23.569904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:23.569867 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a93bd-767f7c6dbc-5k7bx"] Oct 30 13:46:24.005052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:24.005022 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" path="/var/lib/kubelet/pods/b5a8fa96-a0d6-4a71-b7cf-75c02aab471b/volumes" Oct 30 13:46:25.101461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:25.101426 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:26.908578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:26.908544 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:27.127969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:27.127936 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:27.412162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:27.412128 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:28.460173 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:28.460138 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:28.683157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:28.683124 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:28.990731 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:46:28.990691 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:46:28.998589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:28.998564 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:30.122743 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:30.122707 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:30.867649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:30.867608 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:32.833555 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:32.833522 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:34.697006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:34.696974 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:36.395529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:36.395499 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:38.166311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:38.166261 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:39.571992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:39.571963 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:39.790306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:39.762963 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll"] Oct 30 13:46:39.817180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:39.817058 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" podUID="a50c9b9b-2449-49e4-ab44-f03c33a67950" containerName="maas-api" containerID="cri-o://fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82" gracePeriod=30 Oct 30 13:46:39.899786 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:39.899742 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:40.110845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.110804 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:40.124207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.124175 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:46:40.146294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.132933 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls\") pod \"a50c9b9b-2449-49e4-ab44-f03c33a67950\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " Oct 30 13:46:40.146294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.134111 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50c9b9b-2449-49e4-ab44-f03c33a67950-kube-api-access-qrx2s\" (UniqueName: \"kubernetes.io/projected/a50c9b9b-2449-49e4-ab44-f03c33a67950-kube-api-access-qrx2s\") pod \"a50c9b9b-2449-49e4-ab44-f03c33a67950\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " Oct 30 13:46:40.146294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.134225 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50c9b9b-2449-49e4-ab44-f03c33a67950-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9b9b-2449-49e4-ab44-f03c33a67950-openshift-service-ca\") pod \"a50c9b9b-2449-49e4-ab44-f03c33a67950\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " Oct 30 13:46:40.146294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.134325 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-metrics-certs\") pod \"a50c9b9b-2449-49e4-ab44-f03c33a67950\" (UID: \"a50c9b9b-2449-49e4-ab44-f03c33a67950\") " Oct 30 13:46:40.146294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.134975 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50c9b9b-2449-49e4-ab44-f03c33a67950-openshift-service-ca" pod "a50c9b9b-2449-49e4-ab44-f03c33a67950" (UID: "a50c9b9b-2449-49e4-ab44-f03c33a67950"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:46:40.146294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.135706 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9b9b-2449-49e4-ab44-f03c33a67950-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:40.172522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.167482 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls" pod "a50c9b9b-2449-49e4-ab44-f03c33a67950" (UID: "a50c9b9b-2449-49e4-ab44-f03c33a67950"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:46:40.172522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.167571 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-metrics-certs" pod "a50c9b9b-2449-49e4-ab44-f03c33a67950" (UID: "a50c9b9b-2449-49e4-ab44-f03c33a67950"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:46:40.172522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.167683 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50c9b9b-2449-49e4-ab44-f03c33a67950-kube-api-access-qrx2s" pod "a50c9b9b-2449-49e4-ab44-f03c33a67950" (UID: "a50c9b9b-2449-49e4-ab44-f03c33a67950"). InnerVolumeSpecName "kube-api-access-qrx2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:46:40.243366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.243172 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrx2s\" (UniqueName: \"kubernetes.io/projected/a50c9b9b-2449-49e4-ab44-f03c33a67950-kube-api-access-qrx2s\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:40.243366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.243201 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:40.243366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.243213 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a50c9b9b-2449-49e4-ab44-f03c33a67950-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:40.545707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.545672 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:40.651733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.651694 5515 generic.go:356] "Generic (PLEG): container finished" podID="a50c9b9b-2449-49e4-ab44-f03c33a67950" containerID="fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82" exitCode=0 Oct 30 13:46:40.652234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.651763 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" Oct 30 13:46:40.652234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.651829 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" event={"ID":"a50c9b9b-2449-49e4-ab44-f03c33a67950","Type":"ContainerDied","Data":"fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82"} Oct 30 13:46:40.652234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.651898 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll" event={"ID":"a50c9b9b-2449-49e4-ab44-f03c33a67950","Type":"ContainerDied","Data":"a3003cb8ac63c0d5b344f80b997497e268d2eccee07085c39f80cf285ec1403a"} Oct 30 13:46:40.652234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.651929 5515 scope.go:122] "RemoveContainer" containerID="fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82" Oct 30 13:46:40.701903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.701872 5515 scope.go:122] "RemoveContainer" containerID="fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82" Oct 30 13:46:40.702380 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:46:40.702347 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82\": container with ID starting with fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82 not found: ID does not exist" containerID="fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82" Oct 30 13:46:40.702539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.702388 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82"} err="failed to get container status \"fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82\": rpc error: code = NotFound desc = could not find container \"fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82\": container with ID starting with fbba8f129d792a02c0fa07d1b32068afcfc79caeb21b2771cf7f8bfd64356f82 not found: ID does not exist" Oct 30 13:46:40.733464 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.733433 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll"] Oct 30 13:46:40.745533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.745502 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f97234-547dd5d46f-6hrll"] Oct 30 13:46:40.754614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:40.754592 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:41.165657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:41.165626 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:42.003258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:42.003177 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50c9b9b-2449-49e4-ab44-f03c33a67950" path="/var/lib/kubelet/pods/a50c9b9b-2449-49e4-ab44-f03c33a67950/volumes" Oct 30 13:46:42.578089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:42.578050 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:44.210061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:44.210029 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:44.787129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:44.787095 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:46:44.859371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:44.859344 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:45.941293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:45.938821 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:46.955479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:46.955445 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:47.721920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:47.721889 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:48.286161 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:48.286126 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:48.524623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:48.524593 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc"] Oct 30 13:46:48.524853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:48.524806 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" containerID="cri-o://6c36bada41af4544226d789a45f29e84d2d36dda26bb510c4ec7eae91bf8e286" gracePeriod=30 Oct 30 13:46:48.530664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:48.530636 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:48.763905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:48.763867 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:48.977887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:48.977853 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:49.106172 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:49.106085 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg"] Oct 30 13:46:49.106374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:49.106335 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" containerID="cri-o://74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255" gracePeriod=30 Oct 30 13:46:49.204694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:49.204660 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:49.500321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:49.500260 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:49.754122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:49.754030 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:50.019784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:50.019660 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:50.391626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:50.391582 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:50.615629 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:50.615592 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:50.925115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:50.925079 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:51.187769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:51.187694 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:51.930446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:51.930415 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:52.163428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:52.163398 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:53.052887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.052848 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:53.504740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.504703 5515 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:46:53.504946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.504764 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:46:53.755469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.753803 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:53.790330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.790299 5515 generic.go:356] "Generic (PLEG): container finished" podID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerID="6c36bada41af4544226d789a45f29e84d2d36dda26bb510c4ec7eae91bf8e286" exitCode=0 Oct 30 13:46:53.790330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.790316 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" event={"ID":"d9c8d1c1-6e77-4ef4-8372-142e32c0004b","Type":"ContainerDied","Data":"6c36bada41af4544226d789a45f29e84d2d36dda26bb510c4ec7eae91bf8e286"} Oct 30 13:46:53.860274 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.860237 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:46:53.980216 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980179 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-certs\") pod \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " Oct 30 13:46:53.980404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980253 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-credential-socket\") pod \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " Oct 30 13:46:53.980404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980371 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-envoy\") pod \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " Oct 30 13:46:53.980509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980447 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-token\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-token\") pod \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " Oct 30 13:46:53.980559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980499 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-certs" pod "d9c8d1c1-6e77-4ef4-8372-142e32c0004b" (UID: "d9c8d1c1-6e77-4ef4-8372-142e32c0004b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:53.980622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980557 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-data\") pod \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " Oct 30 13:46:53.980622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980592 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-socket\") pod \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " Oct 30 13:46:53.980750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980727 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istiod-ca-cert\") pod \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " Oct 30 13:46:53.980818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980767 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-kube-api-access-hnsdn\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-kube-api-access-hnsdn\") pod \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " Oct 30 13:46:53.980818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980784 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-data" pod "d9c8d1c1-6e77-4ef4-8372-142e32c0004b" (UID: "d9c8d1c1-6e77-4ef4-8372-142e32c0004b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:53.980934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980836 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-credential-socket" pod "d9c8d1c1-6e77-4ef4-8372-142e32c0004b" (UID: "d9c8d1c1-6e77-4ef4-8372-142e32c0004b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:53.980934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980880 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-podinfo\") pod \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\" (UID: \"d9c8d1c1-6e77-4ef4-8372-142e32c0004b\") " Oct 30 13:46:53.981063 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.980961 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-socket" pod "d9c8d1c1-6e77-4ef4-8372-142e32c0004b" (UID: "d9c8d1c1-6e77-4ef4-8372-142e32c0004b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:53.981160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.981113 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istiod-ca-cert" pod "d9c8d1c1-6e77-4ef4-8372-142e32c0004b" (UID: "d9c8d1c1-6e77-4ef4-8372-142e32c0004b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:46:53.981651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.981609 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:53.981651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.981636 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:53.981651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.981653 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:53.981897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.981667 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:53.981897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.981681 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:53.983211 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.983177 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-token" pod "d9c8d1c1-6e77-4ef4-8372-142e32c0004b" (UID: "d9c8d1c1-6e77-4ef4-8372-142e32c0004b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:46:53.983594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.983568 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-podinfo" pod "d9c8d1c1-6e77-4ef4-8372-142e32c0004b" (UID: "d9c8d1c1-6e77-4ef4-8372-142e32c0004b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:46:53.983782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.983756 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-kube-api-access-hnsdn" pod "d9c8d1c1-6e77-4ef4-8372-142e32c0004b" (UID: "d9c8d1c1-6e77-4ef4-8372-142e32c0004b"). InnerVolumeSpecName "kube-api-access-hnsdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:46:53.983864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:53.983767 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-envoy" pod "d9c8d1c1-6e77-4ef4-8372-142e32c0004b" (UID: "d9c8d1c1-6e77-4ef4-8372-142e32c0004b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:54.083342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.083305 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.083342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.083339 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.083342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.083348 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.083818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.083356 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnsdn\" (UniqueName: \"kubernetes.io/projected/d9c8d1c1-6e77-4ef4-8372-142e32c0004b-kube-api-access-hnsdn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.269958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.269871 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:54.378239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.378213 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:46:54.488717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.488682 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-token\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-token\") pod \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " Oct 30 13:46:54.488914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.488787 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-data\") pod \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " Oct 30 13:46:54.488914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.488840 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-envoy\") pod \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " Oct 30 13:46:54.488914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.488875 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istiod-ca-cert\") pod \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " Oct 30 13:46:54.489112 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.489055 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-credential-socket\") pod \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " Oct 30 13:46:54.489180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.489140 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-data" pod "a0dbcfc4-c27b-4955-bfe1-2420985e5687" (UID: "a0dbcfc4-c27b-4955-bfe1-2420985e5687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:54.489180 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.489152 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-podinfo\") pod \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " Oct 30 13:46:54.489912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.489283 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-kube-api-access-tszgn\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-kube-api-access-tszgn\") pod \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " Oct 30 13:46:54.489912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.489323 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-socket\") pod \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " Oct 30 13:46:54.489912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.489398 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-certs\") pod \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\" (UID: \"a0dbcfc4-c27b-4955-bfe1-2420985e5687\") " Oct 30 13:46:54.490106 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.490080 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.490385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.490362 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-certs" pod "a0dbcfc4-c27b-4955-bfe1-2420985e5687" (UID: "a0dbcfc4-c27b-4955-bfe1-2420985e5687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:54.490908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.490878 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-credential-socket" pod "a0dbcfc4-c27b-4955-bfe1-2420985e5687" (UID: "a0dbcfc4-c27b-4955-bfe1-2420985e5687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:54.491148 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.491125 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-socket" pod "a0dbcfc4-c27b-4955-bfe1-2420985e5687" (UID: "a0dbcfc4-c27b-4955-bfe1-2420985e5687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:54.491544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.491519 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istiod-ca-cert" pod "a0dbcfc4-c27b-4955-bfe1-2420985e5687" (UID: "a0dbcfc4-c27b-4955-bfe1-2420985e5687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:46:54.491763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.491714 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-podinfo" pod "a0dbcfc4-c27b-4955-bfe1-2420985e5687" (UID: "a0dbcfc4-c27b-4955-bfe1-2420985e5687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:46:54.491904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.491882 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-token" pod "a0dbcfc4-c27b-4955-bfe1-2420985e5687" (UID: "a0dbcfc4-c27b-4955-bfe1-2420985e5687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:46:54.492521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.492497 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-envoy" pod "a0dbcfc4-c27b-4955-bfe1-2420985e5687" (UID: "a0dbcfc4-c27b-4955-bfe1-2420985e5687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:46:54.493151 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.493121 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-kube-api-access-tszgn" pod "a0dbcfc4-c27b-4955-bfe1-2420985e5687" (UID: "a0dbcfc4-c27b-4955-bfe1-2420985e5687"). InnerVolumeSpecName "kube-api-access-tszgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:46:54.505185 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.505146 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:54.591762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.591728 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.591762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.591761 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.592035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.591776 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tszgn\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-kube-api-access-tszgn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.592035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.591789 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.592035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.591798 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.592035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.591806 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.592035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.591814 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.592035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.591822 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dbcfc4-c27b-4955-bfe1-2420985e5687-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:46:54.783372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.783333 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:54.799589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.799555 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" event={"ID":"d9c8d1c1-6e77-4ef4-8372-142e32c0004b","Type":"ContainerDied","Data":"e521b8444deab1c2e7372b81a9f453e5cf13e37a5e6da7ffa3a8a8e3622345e7"} Oct 30 13:46:54.799589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.799575 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc" Oct 30 13:46:54.799850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.799650 5515 scope.go:122] "RemoveContainer" containerID="6c36bada41af4544226d789a45f29e84d2d36dda26bb510c4ec7eae91bf8e286" Oct 30 13:46:54.801359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.801332 5515 generic.go:356] "Generic (PLEG): container finished" podID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerID="74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255" exitCode=0 Oct 30 13:46:54.801494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.801390 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" Oct 30 13:46:54.801494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.801416 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" event={"ID":"a0dbcfc4-c27b-4955-bfe1-2420985e5687","Type":"ContainerDied","Data":"74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255"} Oct 30 13:46:54.801605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.801494 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg" event={"ID":"a0dbcfc4-c27b-4955-bfe1-2420985e5687","Type":"ContainerDied","Data":"1600d1daa35c54e058c011b6d1ed3899c515c5d894b5ca3f4f7b99b88ea88a94"} Oct 30 13:46:54.817448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.817431 5515 scope.go:122] "RemoveContainer" containerID="74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255" Oct 30 13:46:54.827689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.827664 5515 scope.go:122] "RemoveContainer" containerID="74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255" Oct 30 13:46:54.828009 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:46:54.827988 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255\": container with ID starting with 74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255 not found: ID does not exist" containerID="74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255" Oct 30 13:46:54.828091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.828017 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255"} err="failed to get container status \"74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255\": rpc error: code = NotFound desc = could not find container \"74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255\": container with ID starting with 74c25ad123a18c32836cc0c564aeca73c91c5cb501b46c91edc62ef1d6e7e255 not found: ID does not exist" Oct 30 13:46:54.875496 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.875462 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc"] Oct 30 13:46:54.887651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.887620 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-kjmtc"] Oct 30 13:46:54.934204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.934158 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg"] Oct 30 13:46:54.968173 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:54.968142 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-452bg"] Oct 30 13:46:55.863958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:55.863922 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:56.002758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:56.002729 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" path="/var/lib/kubelet/pods/a0dbcfc4-c27b-4955-bfe1-2420985e5687/volumes" Oct 30 13:46:56.003276 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:56.003253 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" path="/var/lib/kubelet/pods/d9c8d1c1-6e77-4ef4-8372-142e32c0004b/volumes" Oct 30 13:46:56.274501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:56.274469 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:56.499578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:56.499530 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:57.642859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:57.642797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:57.878804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:57.878773 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:58.169323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:58.168902 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:58.992366 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:46:58.992322 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:46:59.118739 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:59.118706 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:59.402534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:59.402493 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:46:59.616664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:46:59.616626 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:00.335402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:00.335364 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:00.569184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:00.569151 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:00.783916 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:00.783884 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:01.602776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:01.602738 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:03.201740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:03.201708 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:05.024792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:05.024758 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:06.680732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:06.680701 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:06.914690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:06.914661 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:08.582476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:08.582439 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:10.241607 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:10.241576 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:11.928621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:11.928591 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:13.120850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:13.120818 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:13.902705 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:13.902663 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:14.200966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:14.200880 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:15.687833 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:15.687799 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:16.192812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:16.192777 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:16.403616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:16.403587 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:17.418001 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:17.417968 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:19.235778 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:19.235743 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:20.782084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:20.782051 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:21.331919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:21.331880 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:23.160435 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:23.160398 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:24.919623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:24.919583 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:26.588774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:26.588739 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:28.204126 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:28.204081 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:28.988888 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:47:28.988848 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:47:29.818503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:29.818450 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:31.544701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:31.544662 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:33.218444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:33.218409 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:34.991618 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:34.991585 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:36.590372 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:36.590338 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:47:36.681468 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:36.681434 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:38.354350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:38.354317 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:40.271904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:40.271868 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:40.487742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:40.487704 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:42.053612 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:42.053571 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:43.855062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:43.855031 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:45.683881 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:45.683852 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:47.336235 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:47.336199 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:49.065201 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:49.065169 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:50.823026 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:50.822988 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:52.607619 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:52.607586 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:54.515393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:54.515363 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:56.347128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:56.347040 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:58.243613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:58.243579 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:47:58.990288 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:47:58.990230 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:47:59.930703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:47:59.930661 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:01.916715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:01.916679 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:04.142899 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:04.142861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:05.869289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:05.869247 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:07.618751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:07.618718 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:09.370738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:09.370704 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:11.093739 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:11.093703 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:12.924446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:12.924359 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:14.780314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:14.780251 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:15.037939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:15.037854 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:15.346159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:15.346064 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:16.335375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:16.335327 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:16.764717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:16.764663 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:17.186100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:17.186069 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:17.434708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:17.434675 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:17.654927 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:17.654896 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:19.039284 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:19.039239 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:20.836013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:20.835979 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:21.923119 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:21.923084 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:23.188824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:23.188786 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:23.398462 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:23.398426 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:23.902080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:23.902038 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:24.665466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:24.665435 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:25.536919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:25.536886 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:25.774338 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:25.774306 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:26.577654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:26.577624 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:28.295040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:28.295001 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:28.987567 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:48:28.987530 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:48:29.881392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:29.881344 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:31.632785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:31.632748 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:32.378780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:32.378749 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:48:33.500654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:33.500624 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:35.236401 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:35.236367 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:36.926985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:36.926955 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:37.250999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:37.250917 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:37.925068 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:37.924987 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:38.172962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:38.172929 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:38.596335 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:38.596305 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:39.221480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:39.221441 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:41.014174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:41.014139 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:42.847853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:42.847822 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:44.766136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:44.766105 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:46.550760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:46.550730 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:46.794757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:46.794727 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:48.698849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:48.698795 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:50.540633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:50.540601 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:50.937888 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:50.937850 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:53.054070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:53.054034 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:54.184327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:54.184291 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:55.069591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:55.069556 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:56.819645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:56.819591 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:57.143840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:57.143809 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:58.900641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:48:58.900611 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:58.989827 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:48:58.989794 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:49:00.576530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:00.576493 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:02.159107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:02.159077 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:02.372040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:02.372001 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:03.822831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:03.822797 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:05.510087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:05.510040 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:07.268155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:07.268120 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:09.035644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:09.035609 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:10.919184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:10.919148 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:12.451819 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:12.451787 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:14.177258 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:14.171087 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:15.958777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:15.958741 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:16.500933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:16.500895 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:17.595802 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:17.595765 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:19.254172 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:19.254133 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:20.838010 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:20.837976 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:21.531219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:21.531183 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:22.735939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:22.735905 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:24.398537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:24.398500 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:26.019523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:26.019481 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:27.703560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:27.703525 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:28.989913 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:49:28.989875 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:49:29.484937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:29.484909 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:29.592964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:29.592933 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:49:31.336885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:31.336839 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:33.315678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:33.315642 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:35.142673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:35.142643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:36.756562 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:36.756526 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:38.388640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:38.388605 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:40.082648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:40.082617 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:41.810011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:41.809976 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:43.493999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:43.493962 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:45.166213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:45.166180 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:46.878454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:46.878418 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:48.464520 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:48.464485 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:48.684535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:48.684495 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:50.233332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:50.233283 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx"] Oct 30 13:49:50.267677 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:50.267645 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz"] Oct 30 13:49:50.304544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:50.304511 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:50.517576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:50.517492 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:50.733139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:50.733106 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:51.991193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:51.991164 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:54.044437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:54.044407 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:55.233670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:55.233627 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" podUID="d841893d-a574-4fb4-bfed-b26bf84074d9" containerName="payload-processing" containerID="cri-o://e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7" gracePeriod=25 Oct 30 13:49:55.268254 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:55.268169 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" podUID="0ee9551b-f70c-48cd-8b53-76883f77cb62" containerName="payload-pre-processing" containerID="cri-o://74384314be4506088906ff3a054c81afc9ba835042f5109b98bb87b6040a9fdd" gracePeriod=25 Oct 30 13:49:56.014161 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:56.014130 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:57.591379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:57.591345 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:58.989041 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:49:58.989000 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:49:59.395232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:49:59.395160 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:00.708706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:00.708668 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" podUID="0ee9551b-f70c-48cd-8b53-76883f77cb62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:50:00.872494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:00.872457 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:01.532730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:01.532692 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:01.715659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:01.715616 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" podUID="d841893d-a574-4fb4-bfed-b26bf84074d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:50:01.814780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:01.814697 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:02.373376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:02.373333 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:03.825801 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:03.825770 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:05.604716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:05.604680 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:07.428090 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:07.428052 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:09.164006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:09.163955 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:10.168146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:10.168092 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:10.407652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:10.407620 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:10.625804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:10.625707 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:10.707857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:10.707826 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" podUID="0ee9551b-f70c-48cd-8b53-76883f77cb62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:50:11.101952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:11.101915 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:11.714887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:11.714854 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" podUID="d841893d-a574-4fb4-bfed-b26bf84074d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:50:12.063720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:12.063557 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25"] Oct 30 13:50:12.075769 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:12.075739 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fbab4a-4kw25"] Oct 30 13:50:13.193726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:13.193689 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:14.003630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:14.003598 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e53cfa-6c99-4543-84ae-be9dbadde34e" path="/var/lib/kubelet/pods/a6e53cfa-6c99-4543-84ae-be9dbadde34e/volumes" Oct 30 13:50:14.900874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:14.900838 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:15.548278 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.548244 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:50:15.569748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.569719 5515 generic.go:356] "Generic (PLEG): container finished" podID="d841893d-a574-4fb4-bfed-b26bf84074d9" containerID="e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7" exitCode=0 Oct 30 13:50:15.569887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.569761 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" Oct 30 13:50:15.569887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.569809 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" event={"ID":"d841893d-a574-4fb4-bfed-b26bf84074d9","Type":"ContainerDied","Data":"e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7"} Oct 30 13:50:15.569887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.569859 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx" event={"ID":"d841893d-a574-4fb4-bfed-b26bf84074d9","Type":"ContainerDied","Data":"6f7e8192bd79fd8d6f68cbb701f99e798b0ca68d1fe281c870c3f8a095c83387"} Oct 30 13:50:15.570008 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.569886 5515 scope.go:122] "RemoveContainer" containerID="e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7" Oct 30 13:50:15.571571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.571546 5515 generic.go:356] "Generic (PLEG): container finished" podID="0ee9551b-f70c-48cd-8b53-76883f77cb62" containerID="74384314be4506088906ff3a054c81afc9ba835042f5109b98bb87b6040a9fdd" exitCode=0 Oct 30 13:50:15.571684 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.571617 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" event={"ID":"0ee9551b-f70c-48cd-8b53-76883f77cb62","Type":"ContainerDied","Data":"74384314be4506088906ff3a054c81afc9ba835042f5109b98bb87b6040a9fdd"} Oct 30 13:50:15.587037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.587024 5515 scope.go:122] "RemoveContainer" containerID="e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7" Oct 30 13:50:15.587333 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:50:15.587311 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7\": container with ID starting with e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7 not found: ID does not exist" containerID="e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7" Oct 30 13:50:15.587431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.587340 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7"} err="failed to get container status \"e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7\": rpc error: code = NotFound desc = could not find container \"e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7\": container with ID starting with e2836b2ac7c48111020a105d977961ad5b392c2bdc95afd660a0c04b5529c8f7 not found: ID does not exist" Oct 30 13:50:15.588870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.588854 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:50:15.720701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.720669 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee9551b-f70c-48cd-8b53-76883f77cb62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee9551b-f70c-48cd-8b53-76883f77cb62-plugins-config-volume\") pod \"0ee9551b-f70c-48cd-8b53-76883f77cb62\" (UID: \"0ee9551b-f70c-48cd-8b53-76883f77cb62\") " Oct 30 13:50:15.720867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.720757 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d841893d-a574-4fb4-bfed-b26bf84074d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841893d-a574-4fb4-bfed-b26bf84074d9-plugins-config-volume\") pod \"d841893d-a574-4fb4-bfed-b26bf84074d9\" (UID: \"d841893d-a574-4fb4-bfed-b26bf84074d9\") " Oct 30 13:50:15.720867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.720827 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d841893d-a574-4fb4-bfed-b26bf84074d9-kube-api-access-j9dkd\" (UniqueName: \"kubernetes.io/projected/d841893d-a574-4fb4-bfed-b26bf84074d9-kube-api-access-j9dkd\") pod \"d841893d-a574-4fb4-bfed-b26bf84074d9\" (UID: \"d841893d-a574-4fb4-bfed-b26bf84074d9\") " Oct 30 13:50:15.720867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.720848 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee9551b-f70c-48cd-8b53-76883f77cb62-kube-api-access-9jh2b\" (UniqueName: \"kubernetes.io/projected/0ee9551b-f70c-48cd-8b53-76883f77cb62-kube-api-access-9jh2b\") pod \"0ee9551b-f70c-48cd-8b53-76883f77cb62\" (UID: \"0ee9551b-f70c-48cd-8b53-76883f77cb62\") " Oct 30 13:50:15.721105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.721079 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee9551b-f70c-48cd-8b53-76883f77cb62-plugins-config-volume" pod "0ee9551b-f70c-48cd-8b53-76883f77cb62" (UID: "0ee9551b-f70c-48cd-8b53-76883f77cb62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:50:15.721193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.721167 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d841893d-a574-4fb4-bfed-b26bf84074d9-plugins-config-volume" pod "d841893d-a574-4fb4-bfed-b26bf84074d9" (UID: "d841893d-a574-4fb4-bfed-b26bf84074d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:50:15.721479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.721455 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee9551b-f70c-48cd-8b53-76883f77cb62-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:15.721599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.721484 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d841893d-a574-4fb4-bfed-b26bf84074d9-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:15.723309 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.723136 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d841893d-a574-4fb4-bfed-b26bf84074d9-kube-api-access-j9dkd" pod "d841893d-a574-4fb4-bfed-b26bf84074d9" (UID: "d841893d-a574-4fb4-bfed-b26bf84074d9"). InnerVolumeSpecName "kube-api-access-j9dkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:50:15.723437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.723372 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee9551b-f70c-48cd-8b53-76883f77cb62-kube-api-access-9jh2b" pod "0ee9551b-f70c-48cd-8b53-76883f77cb62" (UID: "0ee9551b-f70c-48cd-8b53-76883f77cb62"). InnerVolumeSpecName "kube-api-access-9jh2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:50:15.822815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.822725 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9dkd\" (UniqueName: \"kubernetes.io/projected/d841893d-a574-4fb4-bfed-b26bf84074d9-kube-api-access-j9dkd\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:15.822815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.822761 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jh2b\" (UniqueName: \"kubernetes.io/projected/0ee9551b-f70c-48cd-8b53-76883f77cb62-kube-api-access-9jh2b\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:15.919060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.919014 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx"] Oct 30 13:50:15.929696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:15.929662 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0505b0-7d58478d-dqfnx"] Oct 30 13:50:16.003487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:16.003454 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d841893d-a574-4fb4-bfed-b26bf84074d9" path="/var/lib/kubelet/pods/d841893d-a574-4fb4-bfed-b26bf84074d9/volumes" Oct 30 13:50:16.428411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:16.428378 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:50:16.580919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:16.580837 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" Oct 30 13:50:16.580919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:16.580856 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz" event={"ID":"0ee9551b-f70c-48cd-8b53-76883f77cb62","Type":"ContainerDied","Data":"3ff3437012b75aa02760b948e187e110791a194581105782cdbdcabcd98a2dbc"} Oct 30 13:50:16.580919 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:16.580916 5515 scope.go:122] "RemoveContainer" containerID="74384314be4506088906ff3a054c81afc9ba835042f5109b98bb87b6040a9fdd" Oct 30 13:50:16.643984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:16.643955 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz"] Oct 30 13:50:16.644355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:16.644135 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:16.649060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:16.649038 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0505b0-db6f69cc9-ddsdz"] Oct 30 13:50:16.879060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:16.878989 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:18.002574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:18.002538 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee9551b-f70c-48cd-8b53-76883f77cb62" path="/var/lib/kubelet/pods/0ee9551b-f70c-48cd-8b53-76883f77cb62/volumes" Oct 30 13:50:18.414169 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:18.414139 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:19.775262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:19.775223 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:19.989958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:19.989928 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:20.204042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:20.204001 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:21.461368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:21.461322 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc"] Oct 30 13:50:21.689981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:21.689941 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:21.901164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:21.901138 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:22.592939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:22.592906 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:22.828422 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:22.828384 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:23.507105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:23.507070 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:25.381777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:25.381746 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:26.462617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:26.462580 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" podUID="f4f4262a-c772-4592-9224-8a0d4bcb87f4" containerName="payload-pre-processing" containerID="cri-o://8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8" gracePeriod=25 Oct 30 13:50:27.125591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:27.125556 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:27.900774 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:27.900738 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:28.790791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:28.790760 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:28.990901 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:50:28.990858 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:50:30.454661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:30.454629 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:32.235537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:32.235503 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:33.684416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:33.684377 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" podUID="f4f4262a-c772-4592-9224-8a0d4bcb87f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:50:33.948404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:33.948316 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:35.635185 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:35.635152 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:37.281175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:37.281143 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:38.925793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:38.925763 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:40.487929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:40.487898 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:42.166350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:42.166312 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:43.684821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:43.684781 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" podUID="f4f4262a-c772-4592-9224-8a0d4bcb87f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:50:43.833845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:43.833816 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:45.602644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:45.602615 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:46.770628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.770599 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:50:46.821484 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.821433 5515 generic.go:356] "Generic (PLEG): container finished" podID="f4f4262a-c772-4592-9224-8a0d4bcb87f4" containerID="8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8" exitCode=0 Oct 30 13:50:46.821689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.821487 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" Oct 30 13:50:46.821689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.821488 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" event={"ID":"f4f4262a-c772-4592-9224-8a0d4bcb87f4","Type":"ContainerDied","Data":"8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8"} Oct 30 13:50:46.821689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.821626 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc" event={"ID":"f4f4262a-c772-4592-9224-8a0d4bcb87f4","Type":"ContainerDied","Data":"6cb246c5e87b7065fc7cd91ec4e4e46e09ec1b1694a0cc966d9df2dc7cb1ff15"} Oct 30 13:50:46.821689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.821663 5515 scope.go:122] "RemoveContainer" containerID="8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8" Oct 30 13:50:46.835955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.835930 5515 scope.go:122] "RemoveContainer" containerID="8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8" Oct 30 13:50:46.837242 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:50:46.837211 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8\": container with ID starting with 8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8 not found: ID does not exist" containerID="8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8" Oct 30 13:50:46.837474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.837441 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8"} err="failed to get container status \"8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8\": rpc error: code = NotFound desc = could not find container \"8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8\": container with ID starting with 8dfdd3beb94f5678223e7df4ded8a35ee7edb485b1ce73eb39b9ed59e166e0f8 not found: ID does not exist" Oct 30 13:50:46.913476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.913441 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f4262a-c772-4592-9224-8a0d4bcb87f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f4262a-c772-4592-9224-8a0d4bcb87f4-plugins-config-volume\") pod \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\" (UID: \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\") " Oct 30 13:50:46.913724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.913653 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f4262a-c772-4592-9224-8a0d4bcb87f4-kube-api-access-cp6ct\" (UniqueName: \"kubernetes.io/projected/f4f4262a-c772-4592-9224-8a0d4bcb87f4-kube-api-access-cp6ct\") pod \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\" (UID: \"f4f4262a-c772-4592-9224-8a0d4bcb87f4\") " Oct 30 13:50:46.913887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.913864 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f4262a-c772-4592-9224-8a0d4bcb87f4-plugins-config-volume" pod "f4f4262a-c772-4592-9224-8a0d4bcb87f4" (UID: "f4f4262a-c772-4592-9224-8a0d4bcb87f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:50:46.914134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.914116 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f4262a-c772-4592-9224-8a0d4bcb87f4-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:46.915874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:46.915847 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f4262a-c772-4592-9224-8a0d4bcb87f4-kube-api-access-cp6ct" pod "f4f4262a-c772-4592-9224-8a0d4bcb87f4" (UID: "f4f4262a-c772-4592-9224-8a0d4bcb87f4"). InnerVolumeSpecName "kube-api-access-cp6ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:50:47.015644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:47.015554 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp6ct\" (UniqueName: \"kubernetes.io/projected/f4f4262a-c772-4592-9224-8a0d4bcb87f4-kube-api-access-cp6ct\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:47.175740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:47.175696 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc"] Oct 30 13:50:47.187608 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:47.187574 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d9115-dcd4c5b5f-9jmmc"] Oct 30 13:50:47.380828 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:47.380740 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:48.003557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:48.003523 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f4262a-c772-4592-9224-8a0d4bcb87f4" path="/var/lib/kubelet/pods/f4f4262a-c772-4592-9224-8a0d4bcb87f4/volumes" Oct 30 13:50:49.194985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:49.194949 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:50.595089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:50.595050 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:51.136634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:51.136592 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:51.362673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:51.362642 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:51.773442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:51.773395 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:53.111989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:53.111955 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:54.903022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:54.902981 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:55.214473 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:55.214372 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:50:56.314113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.313997 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt"] Oct 30 13:50:56.314985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.314943 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" podUID="c29663ca-5372-4d33-b444-41ad16ff8cb3" containerName="maas-api" containerID="cri-o://7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75" gracePeriod=30 Oct 30 13:50:56.518904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.518868 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:56.659673 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.659640 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:50:56.700933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.700872 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-maas-api-tls\") pod \"c29663ca-5372-4d33-b444-41ad16ff8cb3\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " Oct 30 13:50:56.701206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.701021 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29663ca-5372-4d33-b444-41ad16ff8cb3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29663ca-5372-4d33-b444-41ad16ff8cb3-openshift-service-ca\") pod \"c29663ca-5372-4d33-b444-41ad16ff8cb3\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " Oct 30 13:50:56.701206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.701086 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-metrics-certs\") pod \"c29663ca-5372-4d33-b444-41ad16ff8cb3\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " Oct 30 13:50:56.701206 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.701129 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29663ca-5372-4d33-b444-41ad16ff8cb3-kube-api-access-qhxml\" (UniqueName: \"kubernetes.io/projected/c29663ca-5372-4d33-b444-41ad16ff8cb3-kube-api-access-qhxml\") pod \"c29663ca-5372-4d33-b444-41ad16ff8cb3\" (UID: \"c29663ca-5372-4d33-b444-41ad16ff8cb3\") " Oct 30 13:50:56.702456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.702426 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29663ca-5372-4d33-b444-41ad16ff8cb3-openshift-service-ca" pod "c29663ca-5372-4d33-b444-41ad16ff8cb3" (UID: "c29663ca-5372-4d33-b444-41ad16ff8cb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:50:56.752324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.750209 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-metrics-certs" pod "c29663ca-5372-4d33-b444-41ad16ff8cb3" (UID: "c29663ca-5372-4d33-b444-41ad16ff8cb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:50:56.752324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.752010 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29663ca-5372-4d33-b444-41ad16ff8cb3-kube-api-access-qhxml" pod "c29663ca-5372-4d33-b444-41ad16ff8cb3" (UID: "c29663ca-5372-4d33-b444-41ad16ff8cb3"). InnerVolumeSpecName "kube-api-access-qhxml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:50:56.753447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.753399 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-maas-api-tls" pod "c29663ca-5372-4d33-b444-41ad16ff8cb3" (UID: "c29663ca-5372-4d33-b444-41ad16ff8cb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:50:56.817369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.814863 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:56.817369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.814906 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29663ca-5372-4d33-b444-41ad16ff8cb3-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:56.817369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.814922 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29663ca-5372-4d33-b444-41ad16ff8cb3-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:56.817369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.814937 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhxml\" (UniqueName: \"kubernetes.io/projected/c29663ca-5372-4d33-b444-41ad16ff8cb3-kube-api-access-qhxml\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:50:56.994870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.994832 5515 generic.go:356] "Generic (PLEG): container finished" podID="c29663ca-5372-4d33-b444-41ad16ff8cb3" containerID="7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75" exitCode=0 Oct 30 13:50:56.995051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.994889 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" Oct 30 13:50:56.995051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.994927 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" event={"ID":"c29663ca-5372-4d33-b444-41ad16ff8cb3","Type":"ContainerDied","Data":"7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75"} Oct 30 13:50:56.995051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.994991 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt" event={"ID":"c29663ca-5372-4d33-b444-41ad16ff8cb3","Type":"ContainerDied","Data":"424ac7c4dbc6b14cd61aa4573bad2c8cd7b59b41e5cd79fb8c7551a82df4afda"} Oct 30 13:50:56.995051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:56.995025 5515 scope.go:122] "RemoveContainer" containerID="7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75" Oct 30 13:50:57.007811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:57.007784 5515 scope.go:122] "RemoveContainer" containerID="7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75" Oct 30 13:50:57.008137 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:50:57.008115 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75\": container with ID starting with 7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75 not found: ID does not exist" containerID="7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75" Oct 30 13:50:57.008231 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:57.008145 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75"} err="failed to get container status \"7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75\": rpc error: code = NotFound desc = could not find container \"7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75\": container with ID starting with 7a07d75de0c7ace563c4ee765600baea878a17ceb6aad8e16fad4693af1a6e75 not found: ID does not exist" Oct 30 13:50:57.051732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:57.051697 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt"] Oct 30 13:50:57.055825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:57.055789 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0d9115-67758474c6-fg4tt"] Oct 30 13:50:57.101917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:57.101883 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:57.345348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:57.344795 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:57.781654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:57.781620 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:58.004378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:58.004342 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29663ca-5372-4d33-b444-41ad16ff8cb3" path="/var/lib/kubelet/pods/c29663ca-5372-4d33-b444-41ad16ff8cb3/volumes" Oct 30 13:50:58.059073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:58.058981 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:58.990641 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:50:58.990601 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:50:59.082198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:50:59.082166 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:00.576406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:00.576368 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:00.828892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:00.828802 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:01.043791 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:01.043761 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:02.690087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:02.690054 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:04.327820 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:04.327780 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:04.792683 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:04.792652 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:05.024056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:05.023994 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:05.325753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:05.325717 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:06.274033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:06.274000 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:07.912210 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:07.912174 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:09.494302 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:09.494239 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:11.073333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:11.073254 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:12.712037 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:12.711958 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:14.505224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:14.505192 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:16.249559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:16.249520 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:16.265971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:16.265932 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj"] Oct 30 13:51:16.297089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:16.297050 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng"] Oct 30 13:51:16.519447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:16.511193 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:16.726404 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:16.726368 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:17.895869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:17.895838 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:19.446432 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:19.446399 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:21.095771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:21.095738 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:21.266814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:21.266763 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" podUID="5b299215-d93e-442f-aae4-87fee6eb584e" containerName="payload-processing" containerID="cri-o://30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451" gracePeriod=25 Oct 30 13:51:21.297872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:21.297828 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" podUID="80e0bbee-c23a-4e45-9754-b3c1375e66ff" containerName="payload-pre-processing" containerID="cri-o://ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd" gracePeriod=25 Oct 30 13:51:22.818633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:22.818598 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:23.626621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:23.626590 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:24.572895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:24.572862 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:26.260488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:26.260453 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:26.821763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:26.821720 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" podUID="80e0bbee-c23a-4e45-9754-b3c1375e66ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:51:26.821941 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:26.821735 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" podUID="5b299215-d93e-442f-aae4-87fee6eb584e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:51:27.881248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:27.881215 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:28.989977 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:51:28.989937 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:51:29.416537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:29.416509 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:31.189676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:31.189640 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:32.846935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:32.846902 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:34.565507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:34.565471 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:36.297059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:36.297028 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:36.821916 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:36.821878 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" podUID="5b299215-d93e-442f-aae4-87fee6eb584e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:51:36.822099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:36.821880 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" podUID="80e0bbee-c23a-4e45-9754-b3c1375e66ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:51:38.197367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:38.197330 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:39.911985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:39.911955 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:40.701884 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:40.701846 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:41.599621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.599584 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:41.620328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.620296 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:51:41.649924 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.649896 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:51:41.710388 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.710356 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e0bbee-c23a-4e45-9754-b3c1375e66ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e0bbee-c23a-4e45-9754-b3c1375e66ff-plugins-config-volume\") pod \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\" (UID: \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\") " Oct 30 13:51:41.710593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.710455 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e0bbee-c23a-4e45-9754-b3c1375e66ff-kube-api-access-gcjjv\" (UniqueName: \"kubernetes.io/projected/80e0bbee-c23a-4e45-9754-b3c1375e66ff-kube-api-access-gcjjv\") pod \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\" (UID: \"80e0bbee-c23a-4e45-9754-b3c1375e66ff\") " Oct 30 13:51:41.710593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.710498 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b299215-d93e-442f-aae4-87fee6eb584e-kube-api-access-w48fd\" (UniqueName: \"kubernetes.io/projected/5b299215-d93e-442f-aae4-87fee6eb584e-kube-api-access-w48fd\") pod \"5b299215-d93e-442f-aae4-87fee6eb584e\" (UID: \"5b299215-d93e-442f-aae4-87fee6eb584e\") " Oct 30 13:51:41.710593 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.710577 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b299215-d93e-442f-aae4-87fee6eb584e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b299215-d93e-442f-aae4-87fee6eb584e-plugins-config-volume\") pod \"5b299215-d93e-442f-aae4-87fee6eb584e\" (UID: \"5b299215-d93e-442f-aae4-87fee6eb584e\") " Oct 30 13:51:41.711912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.711877 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b299215-d93e-442f-aae4-87fee6eb584e-plugins-config-volume" pod "5b299215-d93e-442f-aae4-87fee6eb584e" (UID: "5b299215-d93e-442f-aae4-87fee6eb584e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:51:41.712094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.712066 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e0bbee-c23a-4e45-9754-b3c1375e66ff-plugins-config-volume" pod "80e0bbee-c23a-4e45-9754-b3c1375e66ff" (UID: "80e0bbee-c23a-4e45-9754-b3c1375e66ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:51:41.713818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.713788 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e0bbee-c23a-4e45-9754-b3c1375e66ff-kube-api-access-gcjjv" pod "80e0bbee-c23a-4e45-9754-b3c1375e66ff" (UID: "80e0bbee-c23a-4e45-9754-b3c1375e66ff"). InnerVolumeSpecName "kube-api-access-gcjjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:51:41.718578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.718552 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b299215-d93e-442f-aae4-87fee6eb584e-kube-api-access-w48fd" pod "5b299215-d93e-442f-aae4-87fee6eb584e" (UID: "5b299215-d93e-442f-aae4-87fee6eb584e"). InnerVolumeSpecName "kube-api-access-w48fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:51:41.813155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.813116 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcjjv\" (UniqueName: \"kubernetes.io/projected/80e0bbee-c23a-4e45-9754-b3c1375e66ff-kube-api-access-gcjjv\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:51:41.813155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.813151 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48fd\" (UniqueName: \"kubernetes.io/projected/5b299215-d93e-442f-aae4-87fee6eb584e-kube-api-access-w48fd\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:51:41.813155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.813161 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b299215-d93e-442f-aae4-87fee6eb584e-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:51:41.813417 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:41.813171 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e0bbee-c23a-4e45-9754-b3c1375e66ff-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:51:42.383652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.383617 5515 generic.go:356] "Generic (PLEG): container finished" podID="5b299215-d93e-442f-aae4-87fee6eb584e" containerID="30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451" exitCode=0 Oct 30 13:51:42.383839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.383671 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" Oct 30 13:51:42.383839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.383692 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" event={"ID":"5b299215-d93e-442f-aae4-87fee6eb584e","Type":"ContainerDied","Data":"30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451"} Oct 30 13:51:42.383839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.383746 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj" event={"ID":"5b299215-d93e-442f-aae4-87fee6eb584e","Type":"ContainerDied","Data":"cc92e33a5e9f4e27a09bc632043c95cae1488e3511d3b87c7be80cdf1f6f60d6"} Oct 30 13:51:42.383839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.383776 5515 scope.go:122] "RemoveContainer" containerID="30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451" Oct 30 13:51:42.385479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.385453 5515 generic.go:356] "Generic (PLEG): container finished" podID="80e0bbee-c23a-4e45-9754-b3c1375e66ff" containerID="ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd" exitCode=0 Oct 30 13:51:42.385644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.385519 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" event={"ID":"80e0bbee-c23a-4e45-9754-b3c1375e66ff","Type":"ContainerDied","Data":"ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd"} Oct 30 13:51:42.385644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.385560 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" Oct 30 13:51:42.385644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.385575 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng" event={"ID":"80e0bbee-c23a-4e45-9754-b3c1375e66ff","Type":"ContainerDied","Data":"f7fda4a83aedb47db67229124f16ecd49134a55cdb0113b9835b81e3aa12a341"} Oct 30 13:51:42.394613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.394592 5515 scope.go:122] "RemoveContainer" containerID="30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451" Oct 30 13:51:42.394862 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:51:42.394838 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451\": container with ID starting with 30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451 not found: ID does not exist" containerID="30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451" Oct 30 13:51:42.394934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.394869 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451"} err="failed to get container status \"30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451\": rpc error: code = NotFound desc = could not find container \"30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451\": container with ID starting with 30dde8ee00e257879d5ef693f5401aa4fd742f60ca42fadfd9825f2d155f6451 not found: ID does not exist" Oct 30 13:51:42.394934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.394886 5515 scope.go:122] "RemoveContainer" containerID="ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd" Oct 30 13:51:42.404648 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.404432 5515 scope.go:122] "RemoveContainer" containerID="ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd" Oct 30 13:51:42.404767 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:51:42.404745 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd\": container with ID starting with ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd not found: ID does not exist" containerID="ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd" Oct 30 13:51:42.404830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.404778 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd"} err="failed to get container status \"ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd\": rpc error: code = NotFound desc = could not find container \"ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd\": container with ID starting with ab72e816711cca23b5d00d7e8bebdf5edb898c76abc944bb3b5a02af651a71bd not found: ID does not exist" Oct 30 13:51:42.437709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.437677 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng"] Oct 30 13:51:42.450062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.450029 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bd2aaf-599d6f54dc-rn6ng"] Oct 30 13:51:42.476885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.476854 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj"] Oct 30 13:51:42.480591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:42.480568 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bd2aaf-55bb97fcf4-t7bcj"] Oct 30 13:51:43.268543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:43.268506 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:44.006604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:44.006566 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b299215-d93e-442f-aae4-87fee6eb584e" path="/var/lib/kubelet/pods/5b299215-d93e-442f-aae4-87fee6eb584e/volumes" Oct 30 13:51:44.007134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:44.007111 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e0bbee-c23a-4e45-9754-b3c1375e66ff" path="/var/lib/kubelet/pods/80e0bbee-c23a-4e45-9754-b3c1375e66ff/volumes" Oct 30 13:51:44.922716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:44.922682 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:46.496730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:46.496698 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:48.182293 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:48.182233 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:49.400604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.400568 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg"] Oct 30 13:51:49.401134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.400850 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" podUID="59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" containerName="maas-api" containerID="cri-o://d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1" gracePeriod=30 Oct 30 13:51:49.552526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.552494 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:49.714653 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.714620 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:51:49.790283 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.790237 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-maas-api-tls\") pod \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " Oct 30 13:51:49.790460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.790321 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-kube-api-access-f787z\" (UniqueName: \"kubernetes.io/projected/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-kube-api-access-f787z\") pod \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " Oct 30 13:51:49.790460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.790376 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-openshift-service-ca\") pod \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " Oct 30 13:51:49.790557 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.790460 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-metrics-certs\") pod \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\" (UID: \"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc\") " Oct 30 13:51:49.790808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.790782 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-openshift-service-ca" pod "59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" (UID: "59b8ed75-2249-4244-bdfd-29cfa1bfd7dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:51:49.791060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.791037 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:51:49.793185 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.793161 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-maas-api-tls" pod "59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" (UID: "59b8ed75-2249-4244-bdfd-29cfa1bfd7dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:51:49.793311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.793191 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-metrics-certs" pod "59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" (UID: "59b8ed75-2249-4244-bdfd-29cfa1bfd7dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:51:49.793402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.793346 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-kube-api-access-f787z" pod "59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" (UID: "59b8ed75-2249-4244-bdfd-29cfa1bfd7dc"). InnerVolumeSpecName "kube-api-access-f787z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:51:49.919349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.918740 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:51:49.919349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.918779 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f787z\" (UniqueName: \"kubernetes.io/projected/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-kube-api-access-f787z\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:51:49.919349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:49.918795 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:51:50.104239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.104207 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:50.317705 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.317591 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:50.447707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.447671 5515 generic.go:356] "Generic (PLEG): container finished" podID="59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" containerID="d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1" exitCode=0 Oct 30 13:51:50.448205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.447727 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" Oct 30 13:51:50.448205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.447757 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" event={"ID":"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc","Type":"ContainerDied","Data":"d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1"} Oct 30 13:51:50.448205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.447809 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg" event={"ID":"59b8ed75-2249-4244-bdfd-29cfa1bfd7dc","Type":"ContainerDied","Data":"ab5f8855e6340744fa42cfa2803a7a774fe79ef25ebefb9fb022d9e79592be8a"} Oct 30 13:51:50.448205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.447831 5515 scope.go:122] "RemoveContainer" containerID="d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1" Oct 30 13:51:50.458536 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.458518 5515 scope.go:122] "RemoveContainer" containerID="d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1" Oct 30 13:51:50.458817 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:51:50.458800 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1\": container with ID starting with d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1 not found: ID does not exist" containerID="d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1" Oct 30 13:51:50.458858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.458825 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1"} err="failed to get container status \"d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1\": rpc error: code = NotFound desc = could not find container \"d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1\": container with ID starting with d9613cd61c869206f850683fa10ef7815689df26d1a86eb4afd4f5d94a06e2c1 not found: ID does not exist" Oct 30 13:51:50.500796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.500761 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg"] Oct 30 13:51:50.513375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.513341 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bd2aaf-6788bc7976-z8qqg"] Oct 30 13:51:50.875851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:50.875805 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:51.930652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:51.930622 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:52.003780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:52.003747 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" path="/var/lib/kubelet/pods/59b8ed75-2249-4244-bdfd-29cfa1bfd7dc/volumes" Oct 30 13:51:53.599953 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:53.599919 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:55.181856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:55.181819 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:56.903192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:56.903158 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:57.282800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:57.282674 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:57.495844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:57.495803 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:57.743760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:57.743721 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:58.494603 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:58.494559 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:51:58.856721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:51:58.856623 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:58.989366 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:51:58.989327 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:52:00.802958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:00.802919 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:02.687736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:02.687703 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:04.359777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:04.359737 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:04.563886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:04.563855 5515 scope.go:122] "RemoveContainer" containerID="5aaa5fcbfcacc66252a496d43a30d4e2703335c77f69a5516b712ee4bbff226f" Oct 30 13:52:04.574262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:04.574236 5515 scope.go:122] "RemoveContainer" containerID="291bc29e9d5eab5324ea5bbfdd56ab9b7204527a90b1f08848afe9c8a87db58a" Oct 30 13:52:05.995610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:05.995565 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:07.582954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:07.582926 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:08.594861 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:08.594824 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75"] Oct 30 13:52:09.084948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:09.084915 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:09.295248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:09.295212 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:11.153630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.153599 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr"] Oct 30 13:52:11.154128 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.153811 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" podUID="6646659c-fc53-463a-936b-52950d32099e" containerName="discovery" containerID="cri-o://388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2" gracePeriod=30 Oct 30 13:52:11.161690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.161659 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:11.442777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.442753 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 13:52:11.516277 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.516242 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6646659c-fc53-463a-936b-52950d32099e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6646659c-fc53-463a-936b-52950d32099e-istio-csr-ca-configmap\") pod \"6646659c-fc53-463a-936b-52950d32099e\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " Oct 30 13:52:11.516479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.516382 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-csr-dns-cert\") pod \"6646659c-fc53-463a-936b-52950d32099e\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " Oct 30 13:52:11.516479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.516410 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-kube-api-access-klqvb\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-kube-api-access-klqvb\") pod \"6646659c-fc53-463a-936b-52950d32099e\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " Oct 30 13:52:11.516479 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.516471 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6646659c-fc53-463a-936b-52950d32099e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6646659c-fc53-463a-936b-52950d32099e-local-certs\") pod \"6646659c-fc53-463a-936b-52950d32099e\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " Oct 30 13:52:11.516678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.516522 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-kubeconfig\") pod \"6646659c-fc53-463a-936b-52950d32099e\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " Oct 30 13:52:11.516747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.516678 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-istio-token\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-istio-token\") pod \"6646659c-fc53-463a-936b-52950d32099e\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " Oct 30 13:52:11.516747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.516731 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6646659c-fc53-463a-936b-52950d32099e-istio-csr-ca-configmap" pod "6646659c-fc53-463a-936b-52950d32099e" (UID: "6646659c-fc53-463a-936b-52950d32099e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:52:11.516841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.516735 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-cacerts\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-cacerts\") pod \"6646659c-fc53-463a-936b-52950d32099e\" (UID: \"6646659c-fc53-463a-936b-52950d32099e\") " Oct 30 13:52:11.517538 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.517291 5515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6646659c-fc53-463a-936b-52950d32099e-istio-csr-ca-configmap\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:52:11.519449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.519419 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-cacerts" pod "6646659c-fc53-463a-936b-52950d32099e" (UID: "6646659c-fc53-463a-936b-52950d32099e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:52:11.519843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.519807 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-csr-dns-cert" pod "6646659c-fc53-463a-936b-52950d32099e" (UID: "6646659c-fc53-463a-936b-52950d32099e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:52:11.520462 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.520436 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-kubeconfig" pod "6646659c-fc53-463a-936b-52950d32099e" (UID: "6646659c-fc53-463a-936b-52950d32099e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:52:11.520642 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.520618 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-istio-token" pod "6646659c-fc53-463a-936b-52950d32099e" (UID: "6646659c-fc53-463a-936b-52950d32099e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:52:11.520726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.520691 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6646659c-fc53-463a-936b-52950d32099e-local-certs" pod "6646659c-fc53-463a-936b-52950d32099e" (UID: "6646659c-fc53-463a-936b-52950d32099e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:52:11.521238 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.521212 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-kube-api-access-klqvb" pod "6646659c-fc53-463a-936b-52950d32099e" (UID: "6646659c-fc53-463a-936b-52950d32099e"). InnerVolumeSpecName "kube-api-access-klqvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:52:11.598634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.598593 5515 generic.go:356] "Generic (PLEG): container finished" podID="6646659c-fc53-463a-936b-52950d32099e" containerID="388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2" exitCode=0 Oct 30 13:52:11.598845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.598644 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" Oct 30 13:52:11.598845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.598660 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" event={"ID":"6646659c-fc53-463a-936b-52950d32099e","Type":"ContainerDied","Data":"388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2"} Oct 30 13:52:11.598845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.598725 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr" event={"ID":"6646659c-fc53-463a-936b-52950d32099e","Type":"ContainerDied","Data":"a2c4194d0734b3f5926d1522e7d762ea4b26376592c3461e09f4ca2a999c3e8f"} Oct 30 13:52:11.598845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.598754 5515 scope.go:122] "RemoveContainer" containerID="388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2" Oct 30 13:52:11.618393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.618372 5515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-csr-dns-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:52:11.618393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.618393 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klqvb\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-kube-api-access-klqvb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:52:11.618537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.618404 5515 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6646659c-fc53-463a-936b-52950d32099e-local-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:52:11.618537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.618413 5515 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-istio-kubeconfig\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:52:11.618537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.618422 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6646659c-fc53-463a-936b-52950d32099e-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:52:11.618537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.618430 5515 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6646659c-fc53-463a-936b-52950d32099e-cacerts\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:52:11.620230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.620217 5515 scope.go:122] "RemoveContainer" containerID="388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2" Oct 30 13:52:11.620554 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:52:11.620533 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2\": container with ID starting with 388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2 not found: ID does not exist" containerID="388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2" Oct 30 13:52:11.620658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.620560 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2"} err="failed to get container status \"388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2\": rpc error: code = NotFound desc = could not find container \"388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2\": container with ID starting with 388198a62708fa601d741da0d4ce90cfcba322edae07f6a284bfe1ffe595dbc2 not found: ID does not exist" Oct 30 13:52:11.653476 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.653438 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr"] Oct 30 13:52:11.656702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:11.656679 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7qpr"] Oct 30 13:52:12.003055 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:12.003015 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6646659c-fc53-463a-936b-52950d32099e" path="/var/lib/kubelet/pods/6646659c-fc53-463a-936b-52950d32099e/volumes" Oct 30 13:52:12.809657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:12.809628 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:13.595635 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:13.595581 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" podUID="3ef91a41-6516-4710-ae02-3f9dbeac790c" containerName="payload-processing" containerID="cri-o://1b75ebd186e8a7fc4222563c1f0c116a05a75ce3de01aed093ba008278c7a0d0" gracePeriod=25 Oct 30 13:52:14.395136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:14.395098 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:15.964864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:15.964832 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:16.863002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:16.862973 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:17.711965 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:17.711925 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:19.394506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:19.394464 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:20.989400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:20.989371 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:22.652944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:22.652911 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:23.317496 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:23.317459 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" podUID="3ef91a41-6516-4710-ae02-3f9dbeac790c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:52:24.279666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:24.279635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:26.040534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:26.040502 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:27.679658 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:27.679625 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:28.989705 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:52:28.989668 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:52:29.356299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:29.356192 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:30.970416 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:30.970383 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:32.902628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:32.902596 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:33.317663 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:33.317578 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" podUID="3ef91a41-6516-4710-ae02-3f9dbeac790c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:52:33.754621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:33.754592 5515 generic.go:356] "Generic (PLEG): container finished" podID="3ef91a41-6516-4710-ae02-3f9dbeac790c" containerID="1b75ebd186e8a7fc4222563c1f0c116a05a75ce3de01aed093ba008278c7a0d0" exitCode=0 Oct 30 13:52:33.754806 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:33.754673 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" event={"ID":"3ef91a41-6516-4710-ae02-3f9dbeac790c","Type":"ContainerDied","Data":"1b75ebd186e8a7fc4222563c1f0c116a05a75ce3de01aed093ba008278c7a0d0"} Oct 30 13:52:33.886319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:33.886259 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:52:33.914964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:33.914925 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef91a41-6516-4710-ae02-3f9dbeac790c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef91a41-6516-4710-ae02-3f9dbeac790c-plugins-config-volume\") pod \"3ef91a41-6516-4710-ae02-3f9dbeac790c\" (UID: \"3ef91a41-6516-4710-ae02-3f9dbeac790c\") " Oct 30 13:52:33.915401 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:33.915328 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef91a41-6516-4710-ae02-3f9dbeac790c-plugins-config-volume" pod "3ef91a41-6516-4710-ae02-3f9dbeac790c" (UID: "3ef91a41-6516-4710-ae02-3f9dbeac790c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:52:34.016055 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.016020 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef91a41-6516-4710-ae02-3f9dbeac790c-kube-api-access-zmmlm\" (UniqueName: \"kubernetes.io/projected/3ef91a41-6516-4710-ae02-3f9dbeac790c-kube-api-access-zmmlm\") pod \"3ef91a41-6516-4710-ae02-3f9dbeac790c\" (UID: \"3ef91a41-6516-4710-ae02-3f9dbeac790c\") " Oct 30 13:52:34.017751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.017722 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef91a41-6516-4710-ae02-3f9dbeac790c-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:52:34.019227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.018825 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef91a41-6516-4710-ae02-3f9dbeac790c-kube-api-access-zmmlm" pod "3ef91a41-6516-4710-ae02-3f9dbeac790c" (UID: "3ef91a41-6516-4710-ae02-3f9dbeac790c"). InnerVolumeSpecName "kube-api-access-zmmlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:52:34.118874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.118790 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmmlm\" (UniqueName: \"kubernetes.io/projected/3ef91a41-6516-4710-ae02-3f9dbeac790c-kube-api-access-zmmlm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:52:34.691331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.691256 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:34.763585 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.763558 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" Oct 30 13:52:34.763797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.763556 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75" event={"ID":"3ef91a41-6516-4710-ae02-3f9dbeac790c","Type":"ContainerDied","Data":"f640612ac444c19c2fded87dc130e6f01c0413c6a30e681fa0606d8aaf20f889"} Oct 30 13:52:34.763797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.763704 5515 scope.go:122] "RemoveContainer" containerID="1b75ebd186e8a7fc4222563c1f0c116a05a75ce3de01aed093ba008278c7a0d0" Oct 30 13:52:34.817371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.817338 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75"] Oct 30 13:52:34.829501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:34.829468 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71fbeb-7d77b5574-tcp75"] Oct 30 13:52:36.002626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:36.002584 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef91a41-6516-4710-ae02-3f9dbeac790c" path="/var/lib/kubelet/pods/3ef91a41-6516-4710-ae02-3f9dbeac790c/volumes" Oct 30 13:52:36.441864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:36.441829 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:38.114053 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:38.114015 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:39.496516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:39.496480 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:40.082155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:40.082126 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:40.295152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:40.295117 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:40.660328 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:40.660294 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:41.961283 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:41.961234 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:43.527700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:43.527668 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:45.304654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:45.304612 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:47.043571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:47.043527 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:47.686231 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:47.686198 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:47.902855 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:47.902816 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:48.115969 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:48.115882 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:48.509441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:48.509402 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:48.728113 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:48.728073 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:48.944982 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:48.944949 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:50.531032 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:50.530999 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:52.202717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:52.202685 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:53.370318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:53.370281 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt"] Oct 30 13:52:53.588023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:53.587984 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:53.904781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:53.904746 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:54.103821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:54.103781 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:55.739356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:55.739320 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:57.387014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:57.386980 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:58.990450 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:52:58.990413 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:52:59.085885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:59.085853 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529"] Oct 30 13:52:59.123003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:59.122968 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455"] Oct 30 13:52:59.178176 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:59.178141 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:59.394871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:52:59.394838 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:00.841419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:00.841376 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:02.490081 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:02.490044 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:53:02.559690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:02.559643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:04.086933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:04.086883 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" podUID="f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" containerName="payload-processing" containerID="cri-o://f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6" gracePeriod=25 Oct 30 13:53:04.123630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:04.123582 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" podUID="ad4c9f3d-fa30-4061-9ce5-9e557722441f" containerName="payload-pre-processing" containerID="cri-o://c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520" gracePeriod=25 Oct 30 13:53:04.546515 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:04.546481 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:04.714332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:04.714289 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62"] Oct 30 13:53:04.733002 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:04.732968 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht"] Oct 30 13:53:04.946308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:04.946248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:05.164460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:05.164426 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:06.216533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:06.216496 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:07.858931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:07.858892 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:08.391738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.391699 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" containerID="cri-o://b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60" gracePeriod=45 Oct 30 13:53:08.513775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.513748 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11532-5e1c6e"] Oct 30 13:53:08.514519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514504 5515 cpu_manager.go:414] "containerMap: removing container" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerName="cleanup" Oct 30 13:53:08.514566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514521 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerName="cleanup" Oct 30 13:53:08.514566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514535 5515 cpu_manager.go:414] "containerMap: removing container" podUID="27c5eebc-f9da-442c-81b9-831d45fe4c01" containerName="payload-processing" Oct 30 13:53:08.514566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514541 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c5eebc-f9da-442c-81b9-831d45fe4c01" containerName="payload-processing" Oct 30 13:53:08.514566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514551 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c29663ca-5372-4d33-b444-41ad16ff8cb3" containerName="maas-api" Oct 30 13:53:08.514566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514556 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29663ca-5372-4d33-b444-41ad16ff8cb3" containerName="maas-api" Oct 30 13:53:08.514566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514567 5515 cpu_manager.go:414] "containerMap: removing container" podUID="a50c9b9b-2449-49e4-ab44-f03c33a67950" containerName="maas-api" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514572 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50c9b9b-2449-49e4-ab44-f03c33a67950" containerName="maas-api" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514579 5515 cpu_manager.go:414] "containerMap: removing container" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514583 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514593 5515 cpu_manager.go:414] "containerMap: removing container" podUID="47e0ebcf-081f-4027-b015-e48962364b71" containerName="revoke-keys" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514598 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e0ebcf-081f-4027-b015-e48962364b71" containerName="revoke-keys" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514605 5515 cpu_manager.go:414] "containerMap: removing container" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514609 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514617 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f4f4262a-c772-4592-9224-8a0d4bcb87f4" containerName="payload-pre-processing" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514622 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f4262a-c772-4592-9224-8a0d4bcb87f4" containerName="payload-pre-processing" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514629 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514633 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514640 5515 cpu_manager.go:414] "containerMap: removing container" podUID="7c417014-5325-4155-838d-16a0e309477b" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514645 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c417014-5325-4155-838d-16a0e309477b" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514653 5515 cpu_manager.go:414] "containerMap: removing container" podUID="78818086-4780-4d9b-be87-19b549bcdac2" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514657 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78818086-4780-4d9b-be87-19b549bcdac2" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514664 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d841893d-a574-4fb4-bfed-b26bf84074d9" containerName="payload-processing" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514668 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d841893d-a574-4fb4-bfed-b26bf84074d9" containerName="payload-processing" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514677 5515 cpu_manager.go:414] "containerMap: removing container" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514682 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514688 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5b299215-d93e-442f-aae4-87fee6eb584e" containerName="payload-processing" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514693 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b299215-d93e-442f-aae4-87fee6eb584e" containerName="payload-processing" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514699 5515 cpu_manager.go:414] "containerMap: removing container" podUID="59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" containerName="maas-api" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514703 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" containerName="maas-api" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514709 5515 cpu_manager.go:414] "containerMap: removing container" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514713 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514721 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" containerName="payload-pre-processing" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514725 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" containerName="payload-pre-processing" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514735 5515 cpu_manager.go:414] "containerMap: removing container" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514740 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514751 5515 cpu_manager.go:414] "containerMap: removing container" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514756 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514763 5515 cpu_manager.go:414] "containerMap: removing container" podUID="78818086-4780-4d9b-be87-19b549bcdac2" containerName="cleanup" Oct 30 13:53:08.514757 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514767 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78818086-4780-4d9b-be87-19b549bcdac2" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514776 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b36e1b90-f130-4b8d-be62-197d45ef6bc5" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514781 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36e1b90-f130-4b8d-be62-197d45ef6bc5" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514786 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3ef91a41-6516-4710-ae02-3f9dbeac790c" containerName="payload-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514791 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef91a41-6516-4710-ae02-3f9dbeac790c" containerName="payload-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514803 5515 cpu_manager.go:414] "containerMap: removing container" podUID="7c417014-5325-4155-838d-16a0e309477b" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514808 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c417014-5325-4155-838d-16a0e309477b" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514815 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514820 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514827 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c460355f-0748-412a-a6f6-eb0a0548c441" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514832 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c460355f-0748-412a-a6f6-eb0a0548c441" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514838 5515 cpu_manager.go:414] "containerMap: removing container" podUID="80e0bbee-c23a-4e45-9754-b3c1375e66ff" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514843 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e0bbee-c23a-4e45-9754-b3c1375e66ff" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514856 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0ee9551b-f70c-48cd-8b53-76883f77cb62" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514861 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee9551b-f70c-48cd-8b53-76883f77cb62" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514869 5515 cpu_manager.go:414] "containerMap: removing container" podUID="6646659c-fc53-463a-936b-52950d32099e" containerName="discovery" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514872 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6646659c-fc53-463a-936b-52950d32099e" containerName="discovery" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514879 5515 cpu_manager.go:414] "containerMap: removing container" podUID="bf045354-c623-4b61-bbd9-ab57faeb18a2" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514882 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf045354-c623-4b61-bbd9-ab57faeb18a2" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514890 5515 cpu_manager.go:414] "containerMap: removing container" podUID="cc50345e-fefa-4329-a72c-2248c950a603" containerName="main" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514895 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc50345e-fefa-4329-a72c-2248c950a603" containerName="main" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514902 5515 cpu_manager.go:414] "containerMap: removing container" podUID="78818086-4780-4d9b-be87-19b549bcdac2" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514906 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78818086-4780-4d9b-be87-19b549bcdac2" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514913 5515 cpu_manager.go:414] "containerMap: removing container" podUID="8130ae3f-6e4d-428d-a10c-b830ee51e6da" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514917 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8130ae3f-6e4d-428d-a10c-b830ee51e6da" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514925 5515 cpu_manager.go:414] "containerMap: removing container" podUID="aff4dd5f-793d-4148-8be2-8cbe26c2eddf" containerName="payload-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.514930 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff4dd5f-793d-4148-8be2-8cbe26c2eddf" containerName="payload-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515039 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee9551b-f70c-48cd-8b53-76883f77cb62" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515049 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6646659c-fc53-463a-936b-52950d32099e" containerName="discovery" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515055 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf045354-c623-4b61-bbd9-ab57faeb18a2" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515061 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc50345e-fefa-4329-a72c-2248c950a603" containerName="main" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515066 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515073 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="78818086-4780-4d9b-be87-19b549bcdac2" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515078 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8130ae3f-6e4d-428d-a10c-b830ee51e6da" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515086 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff4dd5f-793d-4148-8be2-8cbe26c2eddf" containerName="payload-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515093 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515097 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c5eebc-f9da-442c-81b9-831d45fe4c01" containerName="payload-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515103 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29663ca-5372-4d33-b444-41ad16ff8cb3" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515108 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50c9b9b-2449-49e4-ab44-f03c33a67950" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515116 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e0ebcf-081f-4027-b015-e48962364b71" containerName="revoke-keys" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515121 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a6b23d-d1b5-45f1-bbca-3185a13f5bf3" containerName="istio-proxy" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515127 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f4262a-c772-4592-9224-8a0d4bcb87f4" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515132 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c8d1c1-6e77-4ef4-8372-142e32c0004b" containerName="istio-proxy" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515139 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c417014-5325-4155-838d-16a0e309477b" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515144 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="78818086-4780-4d9b-be87-19b549bcdac2" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515149 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d841893d-a574-4fb4-bfed-b26bf84074d9" containerName="payload-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515155 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515159 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b299215-d93e-442f-aae4-87fee6eb584e" containerName="payload-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515165 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b8ed75-2249-4244-bdfd-29cfa1bfd7dc" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515169 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515175 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a8fa96-a0d6-4a71-b7cf-75c02aab471b" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515183 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dbcfc4-c27b-4955-bfe1-2420985e5687" containerName="istio-proxy" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515188 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36e1b90-f130-4b8d-be62-197d45ef6bc5" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515193 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef91a41-6516-4710-ae02-3f9dbeac790c" containerName="payload-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515199 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c417014-5325-4155-838d-16a0e309477b" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515206 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8adc93-a6b2-4ffe-8c69-08c7ba07789f" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515210 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c417014-5325-4155-838d-16a0e309477b" containerName="cleanup" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515216 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c460355f-0748-412a-a6f6-eb0a0548c441" containerName="maas-api" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515221 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e0bbee-c23a-4e45-9754-b3c1375e66ff" containerName="payload-pre-processing" Oct 30 13:53:08.515830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.515226 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="43855476-25c9-4939-bfc5-3701cf7f04e9" containerName="cleanup" Oct 30 13:53:08.518840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.518823 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-5e1c6e" Oct 30 13:53:08.540067 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.540027 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-5e1c6e"] Oct 30 13:53:08.546495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.546465 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ljkrn\"" Oct 30 13:53:08.620715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.620680 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjlx6\" (UniqueName: \"kubernetes.io/projected/79bbcf52-328c-4076-9ba9-5e70429940cf-kube-api-access-zjlx6\") pod \"test-curl-11532-5e1c6e\" (UID: \"79bbcf52-328c-4076-9ba9-5e70429940cf\") " pod="opendatahub/test-curl-11532-5e1c6e" Oct 30 13:53:08.681420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.681396 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 13:53:08.722043 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.722014 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjlx6\" (UniqueName: \"kubernetes.io/projected/79bbcf52-328c-4076-9ba9-5e70429940cf-kube-api-access-zjlx6\") pod \"test-curl-11532-5e1c6e\" (UID: \"79bbcf52-328c-4076-9ba9-5e70429940cf\") " pod="opendatahub/test-curl-11532-5e1c6e" Oct 30 13:53:08.745762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.745732 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjlx6\" (UniqueName: \"kubernetes.io/projected/79bbcf52-328c-4076-9ba9-5e70429940cf-kube-api-access-zjlx6\") pod \"test-curl-11532-5e1c6e\" (UID: \"79bbcf52-328c-4076-9ba9-5e70429940cf\") " pod="opendatahub/test-curl-11532-5e1c6e" Oct 30 13:53:08.823378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.823349 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-home\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-home\") pod \"7616402d-0c8a-47c1-9f03-fcc54361145b\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " Oct 30 13:53:08.823576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.823479 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-tmp-dir\") pod \"7616402d-0c8a-47c1-9f03-fcc54361145b\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " Oct 30 13:53:08.823576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.823518 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7616402d-0c8a-47c1-9f03-fcc54361145b-kube-api-access-k6x66\" (UniqueName: \"kubernetes.io/projected/7616402d-0c8a-47c1-9f03-fcc54361145b-kube-api-access-k6x66\") pod \"7616402d-0c8a-47c1-9f03-fcc54361145b\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " Oct 30 13:53:08.823576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.823563 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-model-cache\") pod \"7616402d-0c8a-47c1-9f03-fcc54361145b\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " Oct 30 13:53:08.823776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.823604 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-dshm\") pod \"7616402d-0c8a-47c1-9f03-fcc54361145b\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " Oct 30 13:53:08.823776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.823643 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-home" pod "7616402d-0c8a-47c1-9f03-fcc54361145b" (UID: "7616402d-0c8a-47c1-9f03-fcc54361145b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:08.823776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.823662 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7616402d-0c8a-47c1-9f03-fcc54361145b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7616402d-0c8a-47c1-9f03-fcc54361145b-tls-certs\") pod \"7616402d-0c8a-47c1-9f03-fcc54361145b\" (UID: \"7616402d-0c8a-47c1-9f03-fcc54361145b\") " Oct 30 13:53:08.823776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.823750 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-tmp-dir" pod "7616402d-0c8a-47c1-9f03-fcc54361145b" (UID: "7616402d-0c8a-47c1-9f03-fcc54361145b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:08.824005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.823889 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-model-cache" pod "7616402d-0c8a-47c1-9f03-fcc54361145b" (UID: "7616402d-0c8a-47c1-9f03-fcc54361145b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:08.824242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.824215 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:08.824242 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.824244 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:08.824480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.824257 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:08.825951 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.825923 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-dshm" pod "7616402d-0c8a-47c1-9f03-fcc54361145b" (UID: "7616402d-0c8a-47c1-9f03-fcc54361145b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:08.826296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.825983 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7616402d-0c8a-47c1-9f03-fcc54361145b-tls-certs" pod "7616402d-0c8a-47c1-9f03-fcc54361145b" (UID: "7616402d-0c8a-47c1-9f03-fcc54361145b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:53:08.826296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.826024 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7616402d-0c8a-47c1-9f03-fcc54361145b-kube-api-access-k6x66" pod "7616402d-0c8a-47c1-9f03-fcc54361145b" (UID: "7616402d-0c8a-47c1-9f03-fcc54361145b"). InnerVolumeSpecName "kube-api-access-k6x66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:08.832629 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.832605 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-5e1c6e" Oct 30 13:53:08.925964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.925897 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6x66\" (UniqueName: \"kubernetes.io/projected/7616402d-0c8a-47c1-9f03-fcc54361145b-kube-api-access-k6x66\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:08.925964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.925926 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7616402d-0c8a-47c1-9f03-fcc54361145b-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:08.925964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:08.925935 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7616402d-0c8a-47c1-9f03-fcc54361145b-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:09.014064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.014033 5515 generic.go:356] "Generic (PLEG): container finished" podID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerID="b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60" exitCode=0 Oct 30 13:53:09.014248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.014079 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" Oct 30 13:53:09.014248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.014111 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" event={"ID":"7616402d-0c8a-47c1-9f03-fcc54361145b","Type":"ContainerDied","Data":"b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60"} Oct 30 13:53:09.014248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.014149 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt" event={"ID":"7616402d-0c8a-47c1-9f03-fcc54361145b","Type":"ContainerDied","Data":"5170e068b762b893cde84eb7127440d66877f3f03cd178d4dd55909f4b5dee0b"} Oct 30 13:53:09.014248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.014166 5515 scope.go:122] "RemoveContainer" containerID="b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60" Oct 30 13:53:09.024200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.024182 5515 scope.go:122] "RemoveContainer" containerID="b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60" Oct 30 13:53:09.024478 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:09.024458 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60\": container with ID starting with b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60 not found: ID does not exist" containerID="b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60" Oct 30 13:53:09.024540 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.024485 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60"} err="failed to get container status \"b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60\": rpc error: code = NotFound desc = could not find container \"b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60\": container with ID starting with b645e92f2d391fe59178609b3726e185ebde2ad7d30a205d3c4649453e175f60 not found: ID does not exist" Oct 30 13:53:09.068616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.068577 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt"] Oct 30 13:53:09.080486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.080450 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-2-w1-6ff0e6-kserve-794f4df4d6-vxmbt"] Oct 30 13:53:09.139708 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:53:09.139677 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79bbcf52_328c_4076_9ba9_5e70429940cf.slice/crio-36200933dee3269bef879b8c899694279022e4abf51aee072a2451f7a6af1dd6 WatchSource:0}: Error finding container 36200933dee3269bef879b8c899694279022e4abf51aee072a2451f7a6af1dd6: Status 404 returned error can't find the container with id 36200933dee3269bef879b8c899694279022e4abf51aee072a2451f7a6af1dd6 Oct 30 13:53:09.140306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.140247 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-5e1c6e"] Oct 30 13:53:09.628748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.628640 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:09.715690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.715654 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" podUID="a3b8ed37-561a-4a4f-8ad1-74a065862db4" containerName="payload-processing" containerID="cri-o://d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693" gracePeriod=25 Oct 30 13:53:09.734205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:09.734167 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" podUID="0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" containerName="payload-pre-processing" containerID="cri-o://b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33" gracePeriod=25 Oct 30 13:53:10.002475 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:10.002445 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" path="/var/lib/kubelet/pods/7616402d-0c8a-47c1-9f03-fcc54361145b/volumes" Oct 30 13:53:10.023552 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:10.023520 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-5e1c6e" event={"ID":"79bbcf52-328c-4076-9ba9-5e70429940cf","Type":"ContainerStarted","Data":"57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70"} Oct 30 13:53:10.023732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:10.023565 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-5e1c6e" event={"ID":"79bbcf52-328c-4076-9ba9-5e70429940cf","Type":"ContainerStarted","Data":"36200933dee3269bef879b8c899694279022e4abf51aee072a2451f7a6af1dd6"} Oct 30 13:53:10.062200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:10.062146 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11532-5e1c6e" podStartSLOduration=2.062131784 podStartE2EDuration="2.062131784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:10.061527562 +0000 UTC m=+4808.188328461" watchObservedRunningTime="2026-10-30 13:53:10.062131784 +0000 UTC m=+4808.188932683" Oct 30 13:53:10.673847 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:10.673809 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" podUID="f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:53:10.674034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:10.673875 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" podUID="ad4c9f3d-fa30-4061-9ce5-9e557722441f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:53:10.971508 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:10.971426 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11532-5e1c6e"] Oct 30 13:53:10.976670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:10.976638 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11532-5e1c6e"] Oct 30 13:53:11.136752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.136716 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11532-b7e03c"] Oct 30 13:53:11.137492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137476 5515 cpu_manager.go:414] "containerMap: removing container" podUID="7c417014-5325-4155-838d-16a0e309477b" containerName="cleanup" Oct 30 13:53:11.137492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137493 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c417014-5325-4155-838d-16a0e309477b" containerName="cleanup" Oct 30 13:53:11.137604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137522 5515 cpu_manager.go:414] "containerMap: removing container" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerName="cleanup" Oct 30 13:53:11.137604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137527 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerName="cleanup" Oct 30 13:53:11.137604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137538 5515 cpu_manager.go:414] "containerMap: removing container" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerName="cleanup" Oct 30 13:53:11.137604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137543 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerName="cleanup" Oct 30 13:53:11.137604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137553 5515 cpu_manager.go:414] "containerMap: removing container" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" Oct 30 13:53:11.137604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137558 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" Oct 30 13:53:11.137604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137576 5515 cpu_manager.go:414] "containerMap: removing container" podUID="79bbcf52-328c-4076-9ba9-5e70429940cf" containerName="test-curl-11532-5e1c6e" Oct 30 13:53:11.137604 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137581 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bbcf52-328c-4076-9ba9-5e70429940cf" containerName="test-curl-11532-5e1c6e" Oct 30 13:53:11.137845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137676 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a3e642-d0bb-4d0d-8c2b-fbff1bc79a9c" containerName="cleanup" Oct 30 13:53:11.137845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137689 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="78818086-4780-4d9b-be87-19b549bcdac2" containerName="cleanup" Oct 30 13:53:11.137845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137701 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7616402d-0c8a-47c1-9f03-fcc54361145b" containerName="main" Oct 30 13:53:11.137845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137712 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b49827-36bc-480e-b26e-82326bdc8b4b" containerName="cleanup" Oct 30 13:53:11.137845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.137723 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bbcf52-328c-4076-9ba9-5e70429940cf" containerName="test-curl-11532-5e1c6e" Oct 30 13:53:11.141652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.141635 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-b7e03c" Oct 30 13:53:11.162445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.162414 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-b7e03c"] Oct 30 13:53:11.252352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.252240 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n557l\" (UniqueName: \"kubernetes.io/projected/acd5fed9-40f4-47b7-921a-fee6bbcb4667-kube-api-access-n557l\") pod \"test-curl-11532-b7e03c\" (UID: \"acd5fed9-40f4-47b7-921a-fee6bbcb4667\") " pod="opendatahub/test-curl-11532-b7e03c" Oct 30 13:53:11.319410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.319381 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:11.353782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.353747 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n557l\" (UniqueName: \"kubernetes.io/projected/acd5fed9-40f4-47b7-921a-fee6bbcb4667-kube-api-access-n557l\") pod \"test-curl-11532-b7e03c\" (UID: \"acd5fed9-40f4-47b7-921a-fee6bbcb4667\") " pod="opendatahub/test-curl-11532-b7e03c" Oct 30 13:53:11.377031 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.377005 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n557l\" (UniqueName: \"kubernetes.io/projected/acd5fed9-40f4-47b7-921a-fee6bbcb4667-kube-api-access-n557l\") pod \"test-curl-11532-b7e03c\" (UID: \"acd5fed9-40f4-47b7-921a-fee6bbcb4667\") " pod="opendatahub/test-curl-11532-b7e03c" Oct 30 13:53:11.455691 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.455664 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-b7e03c" Oct 30 13:53:11.754349 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:53:11.754318 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacd5fed9_40f4_47b7_921a_fee6bbcb4667.slice/crio-9a757624eadb48de35bc43efea4e62985931a6fc3dbab5ba1ec2d77ea6d42f88 WatchSource:0}: Error finding container 9a757624eadb48de35bc43efea4e62985931a6fc3dbab5ba1ec2d77ea6d42f88: Status 404 returned error can't find the container with id 9a757624eadb48de35bc43efea4e62985931a6fc3dbab5ba1ec2d77ea6d42f88 Oct 30 13:53:11.754501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:11.754444 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-b7e03c"] Oct 30 13:53:12.041312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:12.041218 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-b7e03c" event={"ID":"acd5fed9-40f4-47b7-921a-fee6bbcb4667","Type":"ContainerStarted","Data":"a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26"} Oct 30 13:53:12.041312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:12.041256 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-b7e03c" event={"ID":"acd5fed9-40f4-47b7-921a-fee6bbcb4667","Type":"ContainerStarted","Data":"9a757624eadb48de35bc43efea4e62985931a6fc3dbab5ba1ec2d77ea6d42f88"} Oct 30 13:53:12.041583 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:12.041388 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11532-5e1c6e" podUID="79bbcf52-328c-4076-9ba9-5e70429940cf" containerName="test-curl-11532-5e1c6e" containerID="cri-o://57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70" gracePeriod=2 Oct 30 13:53:12.081959 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:12.081907 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11532-b7e03c" podStartSLOduration=1.081893043 podStartE2EDuration="1.081893043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:12.079317903 +0000 UTC m=+4810.206118800" watchObservedRunningTime="2026-10-30 13:53:12.081893043 +0000 UTC m=+4810.208693941" Oct 30 13:53:12.909760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:12.909726 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:12.967903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:12.967860 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11532-b7e03c"] Oct 30 13:53:12.975428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:12.975396 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11532-b7e03c"] Oct 30 13:53:14.054859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:14.054828 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11532-b7e03c" podUID="acd5fed9-40f4-47b7-921a-fee6bbcb4667" containerName="test-curl-11532-b7e03c" containerID="cri-o://a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26" gracePeriod=2 Oct 30 13:53:14.219832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:14.219808 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-5e1c6e" Oct 30 13:53:14.293230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:14.293191 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bbcf52-328c-4076-9ba9-5e70429940cf-kube-api-access-zjlx6\" (UniqueName: \"kubernetes.io/projected/79bbcf52-328c-4076-9ba9-5e70429940cf-kube-api-access-zjlx6\") pod \"79bbcf52-328c-4076-9ba9-5e70429940cf\" (UID: \"79bbcf52-328c-4076-9ba9-5e70429940cf\") " Oct 30 13:53:14.295370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:14.295340 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bbcf52-328c-4076-9ba9-5e70429940cf-kube-api-access-zjlx6" pod "79bbcf52-328c-4076-9ba9-5e70429940cf" (UID: "79bbcf52-328c-4076-9ba9-5e70429940cf"). InnerVolumeSpecName "kube-api-access-zjlx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:14.394626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:14.394585 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjlx6\" (UniqueName: \"kubernetes.io/projected/79bbcf52-328c-4076-9ba9-5e70429940cf-kube-api-access-zjlx6\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:14.673854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:14.673822 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:15.062374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:15.062287 5515 generic.go:356] "Generic (PLEG): container finished" podID="79bbcf52-328c-4076-9ba9-5e70429940cf" containerID="57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70" exitCode=137 Oct 30 13:53:15.062374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:15.062339 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-5e1c6e" Oct 30 13:53:15.062891 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:15.062381 5515 scope.go:122] "RemoveContainer" containerID="57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70" Oct 30 13:53:15.072935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:15.072914 5515 scope.go:122] "RemoveContainer" containerID="57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70" Oct 30 13:53:15.073231 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:15.073203 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70\": container with ID starting with 57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70 not found: ID does not exist" containerID="57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70" Oct 30 13:53:15.073363 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:15.073237 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70"} err="failed to get container status \"57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70\": rpc error: code = NotFound desc = could not find container \"57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70\": container with ID starting with 57e67c923156cc35daa2f67808354c219947f45f1d0bb12a50ea8a8999dd7f70 not found: ID does not exist" Oct 30 13:53:15.085480 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:15.085448 5515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11532-5e1c6e\" is forbidden: User \"system:node:ip-10-0-134-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-193.ec2.internal' and this object" podUID="79bbcf52-328c-4076-9ba9-5e70429940cf" pod="opendatahub/test-curl-11532-5e1c6e" Oct 30 13:53:15.366787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:15.366693 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" podUID="0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:53:15.366787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:15.366762 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" podUID="a3b8ed37-561a-4a4f-8ad1-74a065862db4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:53:16.002747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:16.002716 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bbcf52-328c-4076-9ba9-5e70429940cf" path="/var/lib/kubelet/pods/79bbcf52-328c-4076-9ba9-5e70429940cf/volumes" Oct 30 13:53:16.216512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:16.216484 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:16.244050 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:16.244025 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-b7e03c" Oct 30 13:53:16.319534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:16.319465 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd5fed9-40f4-47b7-921a-fee6bbcb4667-kube-api-access-n557l\" (UniqueName: \"kubernetes.io/projected/acd5fed9-40f4-47b7-921a-fee6bbcb4667-kube-api-access-n557l\") pod \"acd5fed9-40f4-47b7-921a-fee6bbcb4667\" (UID: \"acd5fed9-40f4-47b7-921a-fee6bbcb4667\") " Oct 30 13:53:16.321546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:16.321522 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd5fed9-40f4-47b7-921a-fee6bbcb4667-kube-api-access-n557l" pod "acd5fed9-40f4-47b7-921a-fee6bbcb4667" (UID: "acd5fed9-40f4-47b7-921a-fee6bbcb4667"). InnerVolumeSpecName "kube-api-access-n557l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:16.421047 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:16.421014 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n557l\" (UniqueName: \"kubernetes.io/projected/acd5fed9-40f4-47b7-921a-fee6bbcb4667-kube-api-access-n557l\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:17.000733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:17.000701 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:17.083640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:17.083603 5515 generic.go:356] "Generic (PLEG): container finished" podID="acd5fed9-40f4-47b7-921a-fee6bbcb4667" containerID="a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26" exitCode=137 Oct 30 13:53:17.083640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:17.083644 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-b7e03c" Oct 30 13:53:17.083885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:17.083705 5515 scope.go:122] "RemoveContainer" containerID="a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26" Oct 30 13:53:17.095179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:17.094987 5515 scope.go:122] "RemoveContainer" containerID="a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26" Oct 30 13:53:17.095307 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:17.095278 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26\": container with ID starting with a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26 not found: ID does not exist" containerID="a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26" Oct 30 13:53:17.095409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:17.095317 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26"} err="failed to get container status \"a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26\": rpc error: code = NotFound desc = could not find container \"a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26\": container with ID starting with a4b92d2607dacc83349bc046fd0294f194a4ae6d054c5285a58ee3e358932f26 not found: ID does not exist" Oct 30 13:53:17.109251 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:17.109224 5515 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11532-b7e03c\" is forbidden: User \"system:node:ip-10-0-134-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-193.ec2.internal' and this object" podUID="acd5fed9-40f4-47b7-921a-fee6bbcb4667" pod="opendatahub/test-curl-11532-b7e03c" Oct 30 13:53:17.932929 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:17.932898 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:18.001935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:18.001895 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd5fed9-40f4-47b7-921a-fee6bbcb4667" path="/var/lib/kubelet/pods/acd5fed9-40f4-47b7-921a-fee6bbcb4667/volumes" Oct 30 13:53:19.718247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:19.718217 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:20.673691 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:20.673648 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" podUID="ad4c9f3d-fa30-4061-9ce5-9e557722441f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:53:20.673872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:20.673713 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" podUID="f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:53:21.389622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:21.389590 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:23.100644 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.100607 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:23.334303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.334255 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt"] Oct 30 13:53:23.335075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.335058 5515 cpu_manager.go:414] "containerMap: removing container" podUID="acd5fed9-40f4-47b7-921a-fee6bbcb4667" containerName="test-curl-11532-b7e03c" Oct 30 13:53:23.335178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.335077 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd5fed9-40f4-47b7-921a-fee6bbcb4667" containerName="test-curl-11532-b7e03c" Oct 30 13:53:23.335178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.335175 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd5fed9-40f4-47b7-921a-fee6bbcb4667" containerName="test-curl-11532-b7e03c" Oct 30 13:53:23.340014 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.339992 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.356342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.356288 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-e9cb4a-openshift-default-dockercfg-5lrtb\"" Oct 30 13:53:23.372831 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.372806 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt"] Oct 30 13:53:23.412208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.412179 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istiod-ca-cert\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.412364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.412241 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-data\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.412417 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.412373 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-envoy\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.412457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.412445 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.412498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.412471 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-token\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.412565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.412547 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkw4h\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-kube-api-access-rkw4h\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.412634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.412619 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-podinfo\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.412679 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.412666 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-certs\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.412715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.412695 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-credential-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514004 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istiod-ca-cert\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514087 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-data\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514154 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-envoy\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514237 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514306 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-token\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514379 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkw4h\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-kube-api-access-rkw4h\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514445 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-podinfo\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514511 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-certs\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514568 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-credential-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514623 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514706 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-data\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514886 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-credential-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514912 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-certs\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.514977 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.514894 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istiod-ca-cert\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.517075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.517043 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-envoy\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.517219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.517193 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-podinfo\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.537894 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.537865 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkw4h\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-kube-api-access-rkw4h\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.538023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.537865 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-token\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.654190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.654156 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:23.969503 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:53:23.969473 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d0d438_e1e1_434a_9b86_abf3f11e7f2b.slice/crio-196fa4becb859185b464f52848771127f36e99095b1257846d5ef3822e7395e9 WatchSource:0}: Error finding container 196fa4becb859185b464f52848771127f36e99095b1257846d5ef3822e7395e9: Status 404 returned error can't find the container with id 196fa4becb859185b464f52848771127f36e99095b1257846d5ef3822e7395e9 Oct 30 13:53:23.970477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.970441 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt"] Oct 30 13:53:23.972530 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.972500 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:53:23.972623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.972564 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:53:23.972623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:23.972593 5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 13:53:24.134950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.134909 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" event={"ID":"04d0d438-e1e1-434a-9b86-abf3f11e7f2b","Type":"ContainerStarted","Data":"6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b"} Oct 30 13:53:24.135406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.135015 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" event={"ID":"04d0d438-e1e1-434a-9b86-abf3f11e7f2b","Type":"ContainerStarted","Data":"196fa4becb859185b464f52848771127f36e99095b1257846d5ef3822e7395e9"} Oct 30 13:53:24.183601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.183541 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" podStartSLOduration=1.183523083 podStartE2EDuration="1.183523083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:24.181301617 +0000 UTC m=+4822.308102516" watchObservedRunningTime="2026-10-30 13:53:24.183523083 +0000 UTC m=+4822.310323986" Oct 30 13:53:24.397858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.397827 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 13:53:24.432241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.432219 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 13:53:24.531381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.531353 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4c9f3d-fa30-4061-9ce5-9e557722441f-kube-api-access-sc8tn\" (UniqueName: \"kubernetes.io/projected/ad4c9f3d-fa30-4061-9ce5-9e557722441f-kube-api-access-sc8tn\") pod \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\" (UID: \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\") " Oct 30 13:53:24.531559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.531396 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-plugins-config-volume\") pod \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\" (UID: \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\") " Oct 30 13:53:24.531559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.531486 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-kube-api-access-tvpdz\" (UniqueName: \"kubernetes.io/projected/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-kube-api-access-tvpdz\") pod \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\" (UID: \"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6\") " Oct 30 13:53:24.531689 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.531624 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4c9f3d-fa30-4061-9ce5-9e557722441f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c9f3d-fa30-4061-9ce5-9e557722441f-plugins-config-volume\") pod \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\" (UID: \"ad4c9f3d-fa30-4061-9ce5-9e557722441f\") " Oct 30 13:53:24.532376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.532350 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4c9f3d-fa30-4061-9ce5-9e557722441f-plugins-config-volume" pod "ad4c9f3d-fa30-4061-9ce5-9e557722441f" (UID: "ad4c9f3d-fa30-4061-9ce5-9e557722441f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:24.532885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.532863 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-plugins-config-volume" pod "f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" (UID: "f69244e3-72b4-4d7d-b277-1ddd0c0e9de6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:24.534616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.534594 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4c9f3d-fa30-4061-9ce5-9e557722441f-kube-api-access-sc8tn" pod "ad4c9f3d-fa30-4061-9ce5-9e557722441f" (UID: "ad4c9f3d-fa30-4061-9ce5-9e557722441f"). InnerVolumeSpecName "kube-api-access-sc8tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:24.535048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.535028 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-kube-api-access-tvpdz" pod "f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" (UID: "f69244e3-72b4-4d7d-b277-1ddd0c0e9de6"). InnerVolumeSpecName "kube-api-access-tvpdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:24.633312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.633247 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4c9f3d-fa30-4061-9ce5-9e557722441f-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:24.633312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.633310 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc8tn\" (UniqueName: \"kubernetes.io/projected/ad4c9f3d-fa30-4061-9ce5-9e557722441f-kube-api-access-sc8tn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:24.633312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.633321 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:24.633516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.633329 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvpdz\" (UniqueName: \"kubernetes.io/projected/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6-kube-api-access-tvpdz\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:24.654771 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.654745 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:24.732123 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:24.732090 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:25.143584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.143552 5515 generic.go:356] "Generic (PLEG): container finished" podID="ad4c9f3d-fa30-4061-9ce5-9e557722441f" containerID="c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520" exitCode=0 Oct 30 13:53:25.144101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.143616 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" Oct 30 13:53:25.144101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.143626 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" event={"ID":"ad4c9f3d-fa30-4061-9ce5-9e557722441f","Type":"ContainerDied","Data":"c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520"} Oct 30 13:53:25.144101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.143681 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455" event={"ID":"ad4c9f3d-fa30-4061-9ce5-9e557722441f","Type":"ContainerDied","Data":"ed7bc46686afc2f2d0be692d60a75deec10aad37bee1427560c6672b2c90aaa5"} Oct 30 13:53:25.144101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.143712 5515 scope.go:122] "RemoveContainer" containerID="c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520" Oct 30 13:53:25.145671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.145641 5515 generic.go:356] "Generic (PLEG): container finished" podID="f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" containerID="f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6" exitCode=0 Oct 30 13:53:25.145671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.145668 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" Oct 30 13:53:25.146837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.146429 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" event={"ID":"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6","Type":"ContainerDied","Data":"f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6"} Oct 30 13:53:25.146837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.146469 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529" event={"ID":"f69244e3-72b4-4d7d-b277-1ddd0c0e9de6","Type":"ContainerDied","Data":"0e14a59c48e771a8df6fb5afb557f5eef94fc14d023287ba0a87ed04780a8732"} Oct 30 13:53:25.155898 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.155879 5515 scope.go:122] "RemoveContainer" containerID="c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520" Oct 30 13:53:25.156167 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:25.156147 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520\": container with ID starting with c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520 not found: ID does not exist" containerID="c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520" Oct 30 13:53:25.156244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.156174 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520"} err="failed to get container status \"c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520\": rpc error: code = NotFound desc = could not find container \"c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520\": container with ID starting with c6aea2d3cc53a3c2d5d273b80e9279b0a7c58569497ef82ecec8e31924246520 not found: ID does not exist" Oct 30 13:53:25.156244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.156191 5515 scope.go:122] "RemoveContainer" containerID="f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6" Oct 30 13:53:25.165731 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.165716 5515 scope.go:122] "RemoveContainer" containerID="f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6" Oct 30 13:53:25.165965 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:25.165948 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6\": container with ID starting with f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6 not found: ID does not exist" containerID="f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6" Oct 30 13:53:25.166022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.165971 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6"} err="failed to get container status \"f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6\": rpc error: code = NotFound desc = could not find container \"f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6\": container with ID starting with f8eba59304f0aded871c27d03f57bd53222320e9dd080411783a8844ebcfbac6 not found: ID does not exist" Oct 30 13:53:25.219779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.219724 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529"] Oct 30 13:53:25.222895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.222865 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6ff0e6-5c9957ff46-bb529"] Oct 30 13:53:25.256513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.256488 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455"] Oct 30 13:53:25.262171 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.262148 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6ff0e6-696f566c6c-x5455"] Oct 30 13:53:25.366577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.366544 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" podUID="0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:53:25.366740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.366547 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" podUID="a3b8ed37-561a-4a4f-8ad1-74a065862db4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:53:25.655896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.655866 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 13:53:25.656065 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:25.655918 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 13:53:26.026005 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:26.025929 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4c9f3d-fa30-4061-9ce5-9e557722441f" path="/var/lib/kubelet/pods/ad4c9f3d-fa30-4061-9ce5-9e557722441f/volumes" Oct 30 13:53:26.026437 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:26.026422 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" path="/var/lib/kubelet/pods/f69244e3-72b4-4d7d-b277-1ddd0c0e9de6/volumes" Oct 30 13:53:26.302503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:26.302398 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:26.655189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:26.655157 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 13:53:26.655383 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:26.655206 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 13:53:27.654885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:27.654850 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 13:53:27.655322 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:27.654896 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 13:53:28.018349 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:28.016172 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:28.654683 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:28.654650 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 13:53:28.654839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:28.654700 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 13:53:28.989617 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:28.989532 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:53:29.655517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:29.655462 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 13:53:29.655867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:29.655820 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 13:53:29.727197 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:29.727165 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:30.074846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.074801 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:30.180508 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.180486 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:53:30.184561 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.184541 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:53:30.226864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.226778 5515 generic.go:356] "Generic (PLEG): container finished" podID="0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" containerID="b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33" exitCode=0 Oct 30 13:53:30.226864 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.226819 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" Oct 30 13:53:30.227073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.226859 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" event={"ID":"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e","Type":"ContainerDied","Data":"b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33"} Oct 30 13:53:30.227073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.226912 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht" event={"ID":"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e","Type":"ContainerDied","Data":"0972458c49615e1144090b73e8c3bf22987020880222f1a416a7df03e7cef19f"} Oct 30 13:53:30.227073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.226939 5515 scope.go:122] "RemoveContainer" containerID="b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33" Oct 30 13:53:30.228442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.228422 5515 generic.go:356] "Generic (PLEG): container finished" podID="a3b8ed37-561a-4a4f-8ad1-74a065862db4" containerID="d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693" exitCode=0 Oct 30 13:53:30.228570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.228468 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" Oct 30 13:53:30.228570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.228487 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" event={"ID":"a3b8ed37-561a-4a4f-8ad1-74a065862db4","Type":"ContainerDied","Data":"d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693"} Oct 30 13:53:30.228570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.228515 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62" event={"ID":"a3b8ed37-561a-4a4f-8ad1-74a065862db4","Type":"ContainerDied","Data":"c247959e582c001d6dc3653bf50ed116090b7c193d46f2c79a96459be714f48a"} Oct 30 13:53:30.237347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.237329 5515 scope.go:122] "RemoveContainer" containerID="b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33" Oct 30 13:53:30.237605 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:30.237587 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33\": container with ID starting with b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33 not found: ID does not exist" containerID="b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33" Oct 30 13:53:30.237696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.237616 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33"} err="failed to get container status \"b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33\": rpc error: code = NotFound desc = could not find container \"b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33\": container with ID starting with b2248ef722c98110f59dd9eded513280f654eb795ea511278c1e4a0e3b38af33 not found: ID does not exist" Oct 30 13:53:30.237696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.237639 5515 scope.go:122] "RemoveContainer" containerID="d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693" Oct 30 13:53:30.246823 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.246800 5515 scope.go:122] "RemoveContainer" containerID="d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693" Oct 30 13:53:30.247136 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:30.247080 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693\": container with ID starting with d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693 not found: ID does not exist" containerID="d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693" Oct 30 13:53:30.247136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.247126 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693"} err="failed to get container status \"d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693\": rpc error: code = NotFound desc = could not find container \"d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693\": container with ID starting with d59960bad6bd057dc7d3be1bc33b80fd054f98624b50167062d42ad83bdb8693 not found: ID does not exist" Oct 30 13:53:30.279147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.279118 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:30.351099 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.351071 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b8ed37-561a-4a4f-8ad1-74a065862db4-kube-api-access-9pvsd\" (UniqueName: \"kubernetes.io/projected/a3b8ed37-561a-4a4f-8ad1-74a065862db4-kube-api-access-9pvsd\") pod \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\" (UID: \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\") " Oct 30 13:53:30.351304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.351114 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b8ed37-561a-4a4f-8ad1-74a065862db4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b8ed37-561a-4a4f-8ad1-74a065862db4-plugins-config-volume\") pod \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\" (UID: \"a3b8ed37-561a-4a4f-8ad1-74a065862db4\") " Oct 30 13:53:30.351304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.351177 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-kube-api-access-zwnxl\" (UniqueName: \"kubernetes.io/projected/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-kube-api-access-zwnxl\") pod \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\" (UID: \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\") " Oct 30 13:53:30.351487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.351466 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-plugins-config-volume\") pod \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\" (UID: \"0dfe827c-ab2e-4d55-9b10-ad2e83dd742e\") " Oct 30 13:53:30.351579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.351555 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b8ed37-561a-4a4f-8ad1-74a065862db4-plugins-config-volume" pod "a3b8ed37-561a-4a4f-8ad1-74a065862db4" (UID: "a3b8ed37-561a-4a4f-8ad1-74a065862db4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:30.351839 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.351816 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-plugins-config-volume" pod "0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" (UID: "0dfe827c-ab2e-4d55-9b10-ad2e83dd742e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:30.352230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.352205 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b8ed37-561a-4a4f-8ad1-74a065862db4-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:30.352230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.352231 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:30.353993 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.353967 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b8ed37-561a-4a4f-8ad1-74a065862db4-kube-api-access-9pvsd" pod "a3b8ed37-561a-4a4f-8ad1-74a065862db4" (UID: "a3b8ed37-561a-4a4f-8ad1-74a065862db4"). InnerVolumeSpecName "kube-api-access-9pvsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:30.354109 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.353975 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-kube-api-access-zwnxl" pod "0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" (UID: "0dfe827c-ab2e-4d55-9b10-ad2e83dd742e"). InnerVolumeSpecName "kube-api-access-zwnxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:30.453952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.453903 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pvsd\" (UniqueName: \"kubernetes.io/projected/a3b8ed37-561a-4a4f-8ad1-74a065862db4-kube-api-access-9pvsd\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:30.453952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.453951 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnxl\" (UniqueName: \"kubernetes.io/projected/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e-kube-api-access-zwnxl\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:30.576084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.576048 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62"] Oct 30 13:53:30.588105 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.588080 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-acf9e2-5f6c7f76b4-slg62"] Oct 30 13:53:30.615350 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.615320 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht"] Oct 30 13:53:30.619981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.619956 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-acf9e2-56f8ddccf5-667ht"] Oct 30 13:53:30.654789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.654758 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Oct 30 13:53:30.654913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:30.654815 5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Oct 30 13:53:31.471308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:31.471243 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:31.659994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:31.659967 5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:31.660502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:31.660473 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:31.661101 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:31.661085 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:53:32.002158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:32.002126 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" path="/var/lib/kubelet/pods/0dfe827c-ab2e-4d55-9b10-ad2e83dd742e/volumes" Oct 30 13:53:32.002634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:32.002620 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b8ed37-561a-4a4f-8ad1-74a065862db4" path="/var/lib/kubelet/pods/a3b8ed37-561a-4a4f-8ad1-74a065862db4/volumes" Oct 30 13:53:32.238789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:32.238731 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:33.421531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:33.421497 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:35.112445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:35.112413 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:36.520637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:36.520602 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:36.953177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:36.953141 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:37.163690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:37.163659 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:37.369643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:37.369560 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:37.743358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:37.743324 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:38.769914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:38.769878 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:38.979179 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:38.979147 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:39.198628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:39.198591 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:40.832624 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:40.832587 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:42.406355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:42.406323 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:44.025207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:44.025175 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:45.448104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:45.448056 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:45.663936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:45.663895 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:45.680569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:45.680529 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd"] Oct 30 13:53:45.681083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:45.681047 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" containerID="cri-o://65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e" gracePeriod=30 Oct 30 13:53:45.891078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:45.891045 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:47.593703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:47.593665 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:49.162857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:49.162820 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:49.968906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:49.968865 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:53:49.969091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:49.968934 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:53:50.823373 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:50.823337 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:50.955390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:50.955333 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:53:51.131046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131015 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-socket\") pod \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " Oct 30 13:53:51.131239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131075 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-podinfo\") pod \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " Oct 30 13:53:51.131239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131096 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-kube-api-access-wtxsp\") pod \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " Oct 30 13:53:51.131239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131139 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-data\") pod \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " Oct 30 13:53:51.131239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131215 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istiod-ca-cert\") pod \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " Oct 30 13:53:51.131528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131256 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-credential-socket\") pod \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " Oct 30 13:53:51.131528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131324 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-socket" pod "c8b4ca2e-e76e-46b0-b514-a03d9827e132" (UID: "c8b4ca2e-e76e-46b0-b514-a03d9827e132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:51.131528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131356 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-envoy\") pod \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " Oct 30 13:53:51.131528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131388 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-certs\") pod \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " Oct 30 13:53:51.131528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131450 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-token\") pod \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\" (UID: \"c8b4ca2e-e76e-46b0-b514-a03d9827e132\") " Oct 30 13:53:51.131799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131577 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-data" pod "c8b4ca2e-e76e-46b0-b514-a03d9827e132" (UID: "c8b4ca2e-e76e-46b0-b514-a03d9827e132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:51.131799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131592 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-credential-socket" pod "c8b4ca2e-e76e-46b0-b514-a03d9827e132" (UID: "c8b4ca2e-e76e-46b0-b514-a03d9827e132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:51.131799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.131745 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-certs" pod "c8b4ca2e-e76e-46b0-b514-a03d9827e132" (UID: "c8b4ca2e-e76e-46b0-b514-a03d9827e132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:51.132149 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.132121 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:51.132306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.132151 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:51.132306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.132166 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:51.132306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.132180 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:51.132306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.132222 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istiod-ca-cert" pod "c8b4ca2e-e76e-46b0-b514-a03d9827e132" (UID: "c8b4ca2e-e76e-46b0-b514-a03d9827e132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:51.133495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.133469 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-podinfo" pod "c8b4ca2e-e76e-46b0-b514-a03d9827e132" (UID: "c8b4ca2e-e76e-46b0-b514-a03d9827e132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:53:51.133582 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.133526 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-envoy" pod "c8b4ca2e-e76e-46b0-b514-a03d9827e132" (UID: "c8b4ca2e-e76e-46b0-b514-a03d9827e132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:53:51.133582 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.133574 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-kube-api-access-wtxsp" pod "c8b4ca2e-e76e-46b0-b514-a03d9827e132" (UID: "c8b4ca2e-e76e-46b0-b514-a03d9827e132"). InnerVolumeSpecName "kube-api-access-wtxsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:51.133732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.133717 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-token" pod "c8b4ca2e-e76e-46b0-b514-a03d9827e132" (UID: "c8b4ca2e-e76e-46b0-b514-a03d9827e132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:51.233077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.233014 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:51.233077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.233041 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:51.233077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.233051 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:51.233077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.233060 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ca2e-e76e-46b0-b514-a03d9827e132-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:51.233077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.233068 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/c8b4ca2e-e76e-46b0-b514-a03d9827e132-kube-api-access-wtxsp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:51.383120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.383088 5515 generic.go:356] "Generic (PLEG): container finished" podID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerID="65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e" exitCode=0 Oct 30 13:53:51.383308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.383140 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" Oct 30 13:53:51.383308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.383168 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" event={"ID":"c8b4ca2e-e76e-46b0-b514-a03d9827e132","Type":"ContainerDied","Data":"65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e"} Oct 30 13:53:51.383308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.383228 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd" event={"ID":"c8b4ca2e-e76e-46b0-b514-a03d9827e132","Type":"ContainerDied","Data":"fea1c9bb0df2ae4b12c648a22f4aa3e985ce40b8190c5ac4a76488c708643f9f"} Oct 30 13:53:51.383308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.383259 5515 scope.go:122] "RemoveContainer" containerID="65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e" Oct 30 13:53:51.394102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.394086 5515 scope.go:122] "RemoveContainer" containerID="2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5" Oct 30 13:53:51.403691 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.403671 5515 scope.go:122] "RemoveContainer" containerID="65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e" Oct 30 13:53:51.403933 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:51.403913 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e\": container with ID starting with 65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e not found: ID does not exist" containerID="65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e" Oct 30 13:53:51.404011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.403947 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e"} err="failed to get container status \"65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e\": rpc error: code = NotFound desc = could not find container \"65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e\": container with ID starting with 65a4f7fcc028b342b350a33e9666ce391bc67cc03ea68743f30516703d62df2e not found: ID does not exist" Oct 30 13:53:51.404011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.403971 5515 scope.go:122] "RemoveContainer" containerID="2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5" Oct 30 13:53:51.404249 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:51.404232 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5\": container with ID starting with 2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5 not found: ID does not exist" containerID="2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5" Oct 30 13:53:51.404364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.404255 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5"} err="failed to get container status \"2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5\": rpc error: code = NotFound desc = could not find container \"2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5\": container with ID starting with 2494932bf7f383125270fff33340d3f1518ed59ee989ef2e0a4ad2a1388066b5 not found: ID does not exist" Oct 30 13:53:51.446844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.446822 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd"] Oct 30 13:53:51.458925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:51.458902 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-acf9e2-openshift-default-5f97bd99d9-jfhbd"] Oct 30 13:53:52.004497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:52.004467 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" path="/var/lib/kubelet/pods/c8b4ca2e-e76e-46b0-b514-a03d9827e132/volumes" Oct 30 13:53:52.588393 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:52.588359 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:54.094725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:54.094690 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:55.656728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:55.656696 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:56.409793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:56.409757 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:56.520960 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:56.520923 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq"] Oct 30 13:53:56.728531 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:56.728411 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:56.940634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:56.940603 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:57.363560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:57.363529 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:58.718093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:58.718059 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r"] Oct 30 13:53:58.718637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:58.718315 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" podUID="07a4146d-bbb6-48c8-b47e-4372fdc387cc" containerName="maas-api" containerID="cri-o://c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c" gracePeriod=30 Oct 30 13:53:58.878952 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:58.878919 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:58.990121 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:58.990051 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 13:53:58.998920 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:58.998887 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:53:59.027447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.027418 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a4146d-bbb6-48c8-b47e-4372fdc387cc-kube-api-access-zb8jb\" (UniqueName: \"kubernetes.io/projected/07a4146d-bbb6-48c8-b47e-4372fdc387cc-kube-api-access-zb8jb\") pod \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " Oct 30 13:53:59.027652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.027478 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a4146d-bbb6-48c8-b47e-4372fdc387cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a4146d-bbb6-48c8-b47e-4372fdc387cc-openshift-service-ca\") pod \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " Oct 30 13:53:59.027652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.027530 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-maas-api-tls\") pod \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " Oct 30 13:53:59.027652 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.027607 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-metrics-certs\") pod \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\" (UID: \"07a4146d-bbb6-48c8-b47e-4372fdc387cc\") " Oct 30 13:53:59.027895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.027866 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a4146d-bbb6-48c8-b47e-4372fdc387cc-openshift-service-ca" pod "07a4146d-bbb6-48c8-b47e-4372fdc387cc" (UID: "07a4146d-bbb6-48c8-b47e-4372fdc387cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:53:59.028213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.028194 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a4146d-bbb6-48c8-b47e-4372fdc387cc-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:59.030098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.030064 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-maas-api-tls" pod "07a4146d-bbb6-48c8-b47e-4372fdc387cc" (UID: "07a4146d-bbb6-48c8-b47e-4372fdc387cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:53:59.030330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.030245 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-metrics-certs" pod "07a4146d-bbb6-48c8-b47e-4372fdc387cc" (UID: "07a4146d-bbb6-48c8-b47e-4372fdc387cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:53:59.030330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.030287 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a4146d-bbb6-48c8-b47e-4372fdc387cc-kube-api-access-zb8jb" pod "07a4146d-bbb6-48c8-b47e-4372fdc387cc" (UID: "07a4146d-bbb6-48c8-b47e-4372fdc387cc"). InnerVolumeSpecName "kube-api-access-zb8jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:53:59.129253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.129216 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:59.129253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.129248 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb8jb\" (UniqueName: \"kubernetes.io/projected/07a4146d-bbb6-48c8-b47e-4372fdc387cc-kube-api-access-zb8jb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:59.129253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.129258 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a4146d-bbb6-48c8-b47e-4372fdc387cc-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:53:59.394029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.393998 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:59.463052 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.463022 5515 generic.go:356] "Generic (PLEG): container finished" podID="07a4146d-bbb6-48c8-b47e-4372fdc387cc" containerID="c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c" exitCode=0 Oct 30 13:53:59.463207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.463074 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" Oct 30 13:53:59.463207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.463101 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" event={"ID":"07a4146d-bbb6-48c8-b47e-4372fdc387cc","Type":"ContainerDied","Data":"c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c"} Oct 30 13:53:59.463207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.463156 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r" event={"ID":"07a4146d-bbb6-48c8-b47e-4372fdc387cc","Type":"ContainerDied","Data":"e5562d0f4226253b976a3df92d29c4001be71830b2baee04cd54730ef05281e9"} Oct 30 13:53:59.463207 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.463178 5515 scope.go:122] "RemoveContainer" containerID="c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c" Oct 30 13:53:59.488761 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.488740 5515 scope.go:122] "RemoveContainer" containerID="c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c" Oct 30 13:53:59.489082 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:53:59.489061 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c\": container with ID starting with c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c not found: ID does not exist" containerID="c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c" Oct 30 13:53:59.489136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.489091 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c"} err="failed to get container status \"c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c\": rpc error: code = NotFound desc = could not find container \"c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c\": container with ID starting with c747b5c5ad56e9e34c6217422e85815a86956cadd7abf5dc4f4161746f8d4f1c not found: ID does not exist" Oct 30 13:53:59.518221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.518183 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r"] Oct 30 13:53:59.530247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.530213 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6ff0e6-9dcfcdc64-7tx9r"] Oct 30 13:53:59.602672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:53:59.602643 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:00.002367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:00.002336 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a4146d-bbb6-48c8-b47e-4372fdc387cc" path="/var/lib/kubelet/pods/07a4146d-bbb6-48c8-b47e-4372fdc387cc/volumes" Oct 30 13:54:00.012783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:00.012759 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:01.175215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:01.175185 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:01.521495 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:01.521414 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" podUID="63cee96e-e0c8-4f3d-8d2d-6df4243d1469" containerName="payload-pre-processing" containerID="cri-o://6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46" gracePeriod=25 Oct 30 13:54:02.618576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:02.618545 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:54:02.856670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:02.856627 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:04.581116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:04.581088 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:05.759708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:05.759671 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:06.583963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:06.583930 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:08.281798 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:08.281767 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:08.436995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:08.436951 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" podUID="63cee96e-e0c8-4f3d-8d2d-6df4243d1469" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:08.511973 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:08.511932 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:08.729672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:08.729635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:10.085360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:10.085325 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:11.699747 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:11.699710 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:13.404717 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:13.404680 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:15.120616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:15.120586 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:16.328021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:16.327986 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf"] Oct 30 13:54:16.362988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:16.362953 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr"] Oct 30 13:54:16.643933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:16.643905 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:16.860716 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:16.860685 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:17.118907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:17.118828 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:17.328560 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:17.328526 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:18.437132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:18.437096 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" podUID="63cee96e-e0c8-4f3d-8d2d-6df4243d1469" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:18.643746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:18.643714 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:19.893572 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:19.893531 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf"] Oct 30 13:54:19.894077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:19.893753 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" podUID="5856c551-dc30-4e8f-a684-74c4e564b3a8" containerName="maas-api" containerID="cri-o://20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a" gracePeriod=30 Oct 30 13:54:20.109307 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.109254 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:20.172356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.163121 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:54:20.239351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.239317 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5856c551-dc30-4e8f-a684-74c4e564b3a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5856c551-dc30-4e8f-a684-74c4e564b3a8-openshift-service-ca\") pod \"5856c551-dc30-4e8f-a684-74c4e564b3a8\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " Oct 30 13:54:20.239536 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.239503 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-metrics-certs\") pod \"5856c551-dc30-4e8f-a684-74c4e564b3a8\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " Oct 30 13:54:20.239625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.239604 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-maas-api-tls\") pod \"5856c551-dc30-4e8f-a684-74c4e564b3a8\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " Oct 30 13:54:20.239938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.239758 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5856c551-dc30-4e8f-a684-74c4e564b3a8-kube-api-access-6djv4\" (UniqueName: \"kubernetes.io/projected/5856c551-dc30-4e8f-a684-74c4e564b3a8-kube-api-access-6djv4\") pod \"5856c551-dc30-4e8f-a684-74c4e564b3a8\" (UID: \"5856c551-dc30-4e8f-a684-74c4e564b3a8\") " Oct 30 13:54:20.240182 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.239759 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5856c551-dc30-4e8f-a684-74c4e564b3a8-openshift-service-ca" pod "5856c551-dc30-4e8f-a684-74c4e564b3a8" (UID: "5856c551-dc30-4e8f-a684-74c4e564b3a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:54:20.240681 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.240654 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5856c551-dc30-4e8f-a684-74c4e564b3a8-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:20.243701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.242213 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-metrics-certs" pod "5856c551-dc30-4e8f-a684-74c4e564b3a8" (UID: "5856c551-dc30-4e8f-a684-74c4e564b3a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:54:20.243701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.242371 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5856c551-dc30-4e8f-a684-74c4e564b3a8-kube-api-access-6djv4" pod "5856c551-dc30-4e8f-a684-74c4e564b3a8" (UID: "5856c551-dc30-4e8f-a684-74c4e564b3a8"). InnerVolumeSpecName "kube-api-access-6djv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:54:20.243997 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.243971 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-maas-api-tls" pod "5856c551-dc30-4e8f-a684-74c4e564b3a8" (UID: "5856c551-dc30-4e8f-a684-74c4e564b3a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:54:20.342056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.342025 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:20.342056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.342053 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5856c551-dc30-4e8f-a684-74c4e564b3a8-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:20.342056 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.342063 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6djv4\" (UniqueName: \"kubernetes.io/projected/5856c551-dc30-4e8f-a684-74c4e564b3a8-kube-api-access-6djv4\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:20.580664 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.580583 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:20.607625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.607592 5515 generic.go:356] "Generic (PLEG): container finished" podID="5856c551-dc30-4e8f-a684-74c4e564b3a8" containerID="20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a" exitCode=0 Oct 30 13:54:20.607808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.607635 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" event={"ID":"5856c551-dc30-4e8f-a684-74c4e564b3a8","Type":"ContainerDied","Data":"20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a"} Oct 30 13:54:20.607808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.607642 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" Oct 30 13:54:20.607808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.607686 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf" event={"ID":"5856c551-dc30-4e8f-a684-74c4e564b3a8","Type":"ContainerDied","Data":"8964caea71509a40419f058f1fa6e3ea80f759b55c2c40dea6274d5e2f3a752d"} Oct 30 13:54:20.607808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.607714 5515 scope.go:122] "RemoveContainer" containerID="20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a" Oct 30 13:54:20.629502 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.629477 5515 scope.go:122] "RemoveContainer" containerID="20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a" Oct 30 13:54:20.629773 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:54:20.629755 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a\": container with ID starting with 20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a not found: ID does not exist" containerID="20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a" Oct 30 13:54:20.629838 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.629782 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a"} err="failed to get container status \"20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a\": rpc error: code = NotFound desc = could not find container \"20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a\": container with ID starting with 20561d6047ae925a3272b6e64e38f529016f0a1d0ba0051d65a8bdb5f307079a not found: ID does not exist" Oct 30 13:54:20.661767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.661731 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf"] Oct 30 13:54:20.674194 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.674156 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-88e9f3-7d9bb67bf7-jrjlf"] Oct 30 13:54:20.780034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:20.780004 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:21.216670 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.216642 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:21.329229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.329195 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" podUID="85074b34-4622-4d92-a481-2075120a066c" containerName="payload-processing" containerID="cri-o://a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140" gracePeriod=25 Oct 30 13:54:21.364226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.364197 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" podUID="f1c02cc8-b166-4d7a-911d-1f731e97f461" containerName="payload-pre-processing" containerID="cri-o://1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b" gracePeriod=25 Oct 30 13:54:21.804156 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.804129 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:54:21.862379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.862349 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-plugins-config-volume\") pod \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\" (UID: \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\") " Oct 30 13:54:21.862551 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.862452 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-kube-api-access-7szc5\" (UniqueName: \"kubernetes.io/projected/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-kube-api-access-7szc5\") pod \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\" (UID: \"63cee96e-e0c8-4f3d-8d2d-6df4243d1469\") " Oct 30 13:54:21.862832 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.862798 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-plugins-config-volume" pod "63cee96e-e0c8-4f3d-8d2d-6df4243d1469" (UID: "63cee96e-e0c8-4f3d-8d2d-6df4243d1469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:54:21.864671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.864641 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-kube-api-access-7szc5" pod "63cee96e-e0c8-4f3d-8d2d-6df4243d1469" (UID: "63cee96e-e0c8-4f3d-8d2d-6df4243d1469"). InnerVolumeSpecName "kube-api-access-7szc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:54:21.964522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.964494 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szc5\" (UniqueName: \"kubernetes.io/projected/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-kube-api-access-7szc5\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:21.964522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:21.964520 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cee96e-e0c8-4f3d-8d2d-6df4243d1469-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:22.002257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.002226 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5856c551-dc30-4e8f-a684-74c4e564b3a8" path="/var/lib/kubelet/pods/5856c551-dc30-4e8f-a684-74c4e564b3a8/volumes" Oct 30 13:54:22.177347 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.177317 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:22.623070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.622972 5515 generic.go:356] "Generic (PLEG): container finished" podID="63cee96e-e0c8-4f3d-8d2d-6df4243d1469" containerID="6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46" exitCode=0 Oct 30 13:54:22.623070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.623022 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" Oct 30 13:54:22.623070 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.623052 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" event={"ID":"63cee96e-e0c8-4f3d-8d2d-6df4243d1469","Type":"ContainerDied","Data":"6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46"} Oct 30 13:54:22.623636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.623125 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq" event={"ID":"63cee96e-e0c8-4f3d-8d2d-6df4243d1469","Type":"ContainerDied","Data":"de24baeae41f993c33bb26d1088e0bc65bd0379e28fb1d176325bff8df133d67"} Oct 30 13:54:22.623636 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.623157 5515 scope.go:122] "RemoveContainer" containerID="6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46" Oct 30 13:54:22.636436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.636414 5515 scope.go:122] "RemoveContainer" containerID="6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46" Oct 30 13:54:22.636769 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:54:22.636747 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46\": container with ID starting with 6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46 not found: ID does not exist" containerID="6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46" Oct 30 13:54:22.636846 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.636784 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46"} err="failed to get container status \"6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46\": rpc error: code = NotFound desc = could not find container \"6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46\": container with ID starting with 6c4dfed8863eb2133d8f6a44bb5ff06b69ff975c7c105d9f1cb3335c2d08be46 not found: ID does not exist" Oct 30 13:54:22.677734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.677708 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq"] Oct 30 13:54:22.688571 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.688546 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:22.688958 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.688938 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-88e9f3-76786c4475-bg4xq"] Oct 30 13:54:22.895634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:22.895601 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:23.322880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:23.322786 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:24.002597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:24.002562 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63cee96e-e0c8-4f3d-8d2d-6df4243d1469" path="/var/lib/kubelet/pods/63cee96e-e0c8-4f3d-8d2d-6df4243d1469/volumes" Oct 30 13:54:24.601083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:24.601050 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:26.291869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:26.291831 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:27.329399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:27.329363 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" podUID="f1c02cc8-b166-4d7a-911d-1f731e97f461" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:27.329399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:27.329379 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" podUID="85074b34-4622-4d92-a481-2075120a066c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:28.059042 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:28.059004 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:28.412352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:28.412316 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:28.648503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:28.648470 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:28.862686 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:28.862597 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:28.988324 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:54:28.988259 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:54:29.846854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:29.846823 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:30.812821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:30.812788 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:31.021486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:31.021448 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:31.259152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:31.259116 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:31.515775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:31.515681 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:33.448674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:33.448635 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:34.216015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:34.215985 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:35.305314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:35.305262 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:37.161737 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:37.161698 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:37.330015 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:37.329975 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" podUID="85074b34-4622-4d92-a481-2075120a066c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:37.330189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:37.329975 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" podUID="f1c02cc8-b166-4d7a-911d-1f731e97f461" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:37.458024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:37.457952 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:37.661344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:37.661313 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:38.809352 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:38.809319 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:39.745671 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:39.745635 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx"] Oct 30 13:54:39.779390 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:39.779360 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk"] Oct 30 13:54:39.998843 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:39.998741 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:40.216311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:40.216277 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:40.508290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:40.508183 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:41.645127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.645103 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:54:41.670071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.670047 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:54:41.759115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.759052 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:41.787310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.786872 5515 generic.go:356] "Generic (PLEG): container finished" podID="85074b34-4622-4d92-a481-2075120a066c" containerID="a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140" exitCode=0 Oct 30 13:54:41.787310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.786988 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" event={"ID":"85074b34-4622-4d92-a481-2075120a066c","Type":"ContainerDied","Data":"a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140"} Oct 30 13:54:41.787310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.787034 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" event={"ID":"85074b34-4622-4d92-a481-2075120a066c","Type":"ContainerDied","Data":"1f3723220472d2d416da6150c8f4217eaa8c696f931c2d9083cf714cd05f8177"} Oct 30 13:54:41.787310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.787062 5515 scope.go:122] "RemoveContainer" containerID="a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140" Oct 30 13:54:41.787310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.787223 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf" Oct 30 13:54:41.790783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.790755 5515 generic.go:356] "Generic (PLEG): container finished" podID="f1c02cc8-b166-4d7a-911d-1f731e97f461" containerID="1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b" exitCode=0 Oct 30 13:54:41.796480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.790862 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" event={"ID":"f1c02cc8-b166-4d7a-911d-1f731e97f461","Type":"ContainerDied","Data":"1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b"} Oct 30 13:54:41.796480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.790899 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" event={"ID":"f1c02cc8-b166-4d7a-911d-1f731e97f461","Type":"ContainerDied","Data":"fb81992713cef80a35bc853e3b1fa194be9bdc01aea4e84d95e1c91e3a801738"} Oct 30 13:54:41.796480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.791201 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr" Oct 30 13:54:41.803075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.802930 5515 scope.go:122] "RemoveContainer" containerID="a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140" Oct 30 13:54:41.803651 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:54:41.803481 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140\": container with ID starting with a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140 not found: ID does not exist" containerID="a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140" Oct 30 13:54:41.803651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.803520 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140"} err="failed to get container status \"a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140\": rpc error: code = NotFound desc = could not find container \"a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140\": container with ID starting with a8fb3510961f749e453f96a033fc263fec4704c2c8cf4e2fcf85eefbe4a8d140 not found: ID does not exist" Oct 30 13:54:41.803651 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.803550 5515 scope.go:122] "RemoveContainer" containerID="1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b" Oct 30 13:54:41.815750 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.815731 5515 scope.go:122] "RemoveContainer" containerID="1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b" Oct 30 13:54:41.816075 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:54:41.816047 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b\": container with ID starting with 1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b not found: ID does not exist" containerID="1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b" Oct 30 13:54:41.816220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.816086 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b"} err="failed to get container status \"1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b\": rpc error: code = NotFound desc = could not find container \"1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b\": container with ID starting with 1cec5ab6e7e77baa279a2a771b429f5ec2bb6b3f32162f4dcabb1f15c540668b not found: ID does not exist" Oct 30 13:54:41.826397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.826368 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85074b34-4622-4d92-a481-2075120a066c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85074b34-4622-4d92-a481-2075120a066c-plugins-config-volume\") pod \"85074b34-4622-4d92-a481-2075120a066c\" (UID: \"85074b34-4622-4d92-a481-2075120a066c\") " Oct 30 13:54:41.826524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.826430 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c02cc8-b166-4d7a-911d-1f731e97f461-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c02cc8-b166-4d7a-911d-1f731e97f461-plugins-config-volume\") pod \"f1c02cc8-b166-4d7a-911d-1f731e97f461\" (UID: \"f1c02cc8-b166-4d7a-911d-1f731e97f461\") " Oct 30 13:54:41.826601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.826534 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c02cc8-b166-4d7a-911d-1f731e97f461-kube-api-access-c9hr8\" (UniqueName: \"kubernetes.io/projected/f1c02cc8-b166-4d7a-911d-1f731e97f461-kube-api-access-c9hr8\") pod \"f1c02cc8-b166-4d7a-911d-1f731e97f461\" (UID: \"f1c02cc8-b166-4d7a-911d-1f731e97f461\") " Oct 30 13:54:41.826601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.826568 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85074b34-4622-4d92-a481-2075120a066c-kube-api-access-qkbs4\" (UniqueName: \"kubernetes.io/projected/85074b34-4622-4d92-a481-2075120a066c-kube-api-access-qkbs4\") pod \"85074b34-4622-4d92-a481-2075120a066c\" (UID: \"85074b34-4622-4d92-a481-2075120a066c\") " Oct 30 13:54:41.826796 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.826776 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85074b34-4622-4d92-a481-2075120a066c-plugins-config-volume" pod "85074b34-4622-4d92-a481-2075120a066c" (UID: "85074b34-4622-4d92-a481-2075120a066c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:54:41.827127 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.827107 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c02cc8-b166-4d7a-911d-1f731e97f461-plugins-config-volume" pod "f1c02cc8-b166-4d7a-911d-1f731e97f461" (UID: "f1c02cc8-b166-4d7a-911d-1f731e97f461"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:54:41.827257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.827244 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85074b34-4622-4d92-a481-2075120a066c-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:41.827428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.827259 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c02cc8-b166-4d7a-911d-1f731e97f461-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:41.829391 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.829358 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c02cc8-b166-4d7a-911d-1f731e97f461-kube-api-access-c9hr8" pod "f1c02cc8-b166-4d7a-911d-1f731e97f461" (UID: "f1c02cc8-b166-4d7a-911d-1f731e97f461"). InnerVolumeSpecName "kube-api-access-c9hr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:54:41.829605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.829580 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85074b34-4622-4d92-a481-2075120a066c-kube-api-access-qkbs4" pod "85074b34-4622-4d92-a481-2075120a066c" (UID: "85074b34-4622-4d92-a481-2075120a066c"). InnerVolumeSpecName "kube-api-access-qkbs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:54:41.929262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.929228 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9hr8\" (UniqueName: \"kubernetes.io/projected/f1c02cc8-b166-4d7a-911d-1f731e97f461-kube-api-access-c9hr8\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:41.929262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:41.929261 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkbs4\" (UniqueName: \"kubernetes.io/projected/85074b34-4622-4d92-a481-2075120a066c-kube-api-access-qkbs4\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:54:42.128856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:42.127702 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf"] Oct 30 13:54:42.147918 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:42.147860 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-63499e-5dd54b859b-c22tf"] Oct 30 13:54:42.170970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:42.170936 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr"] Oct 30 13:54:42.174659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:42.174616 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-63499e-85956dd9bc-c2llr"] Oct 30 13:54:42.408558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:42.408527 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:42.694780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:42.694741 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579"] Oct 30 13:54:42.726220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:42.726195 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5"] Oct 30 13:54:42.937985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:42.937946 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:43.145006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:43.144972 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:44.002987 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:44.002953 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85074b34-4622-4d92-a481-2075120a066c" path="/var/lib/kubelet/pods/85074b34-4622-4d92-a481-2075120a066c/volumes" Oct 30 13:54:44.003517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:44.003501 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c02cc8-b166-4d7a-911d-1f731e97f461" path="/var/lib/kubelet/pods/f1c02cc8-b166-4d7a-911d-1f731e97f461/volumes" Oct 30 13:54:44.214783 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:44.214754 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:44.746662 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:44.746621 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" podUID="fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" containerName="payload-processing" containerID="cri-o://10c9716038d30923c0d8cfc46b1e964fb1c994ed7a823a3323e1a9242944479e" gracePeriod=25 Oct 30 13:54:44.780591 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:44.780559 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" podUID="ea2cbef4-f022-4196-afac-643eda0f7855" containerName="payload-pre-processing" containerID="cri-o://ea78a51d9f07b96f4c3173e946a87d43bd5d8b59c5e543873b335fa5ddae7ded" gracePeriod=25 Oct 30 13:54:45.989025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:45.988990 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:47.620089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:47.620045 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:47.695277 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:47.695184 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" podUID="fa629737-3309-43f6-8d6d-bea2f57918de" containerName="payload-processing" containerID="cri-o://34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075" gracePeriod=25 Oct 30 13:54:47.727086 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:47.727049 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" podUID="488a7efe-889c-4c6d-a9a1-5fee8a394212" containerName="payload-pre-processing" containerID="cri-o://5ff27a188a9b594686a406dba028f2c3cef939d51c878141133b28d57d3c3767" gracePeriod=25 Oct 30 13:54:49.212377 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:49.212338 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" podUID="fa629737-3309-43f6-8d6d-bea2f57918de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:49.305568 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:49.305534 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:50.218943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:50.218900 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" podUID="488a7efe-889c-4c6d-a9a1-5fee8a394212" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:50.996365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:50.996330 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:52.585566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:52.585530 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:52.792508 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:52.792479 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:53.556312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:53.556245 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" podUID="ea2cbef4-f022-4196-afac-643eda0f7855" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:54.269009 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:54.268976 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:54.582397 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:54.582308 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" podUID="fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:55.925117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:55.925087 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:56.539844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:56.539815 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:57.680414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:57.680383 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:58.988082 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:54:58.988042 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:54:59.022114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:59.022085 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:59.213048 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:59.213014 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" podUID="fa629737-3309-43f6-8d6d-bea2f57918de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:54:59.542428 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:59.542395 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:59.745285 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:54:59.745246 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:00.219533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:00.219489 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" podUID="488a7efe-889c-4c6d-a9a1-5fee8a394212" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:55:00.223913 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:00.223878 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:01.156513 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.156477 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6"] Oct 30 13:55:01.158094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.158059 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" podUID="9a1973c8-eb1b-471e-a568-b81943e66597" containerName="maas-api" containerID="cri-o://53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8" gracePeriod=30 Oct 30 13:55:01.318482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.318446 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:01.440239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.440164 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:55:01.571253 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.571220 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1973c8-eb1b-471e-a568-b81943e66597-kube-api-access-czlvw\" (UniqueName: \"kubernetes.io/projected/9a1973c8-eb1b-471e-a568-b81943e66597-kube-api-access-czlvw\") pod \"9a1973c8-eb1b-471e-a568-b81943e66597\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " Oct 30 13:55:01.571455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.571324 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-maas-api-tls\") pod \"9a1973c8-eb1b-471e-a568-b81943e66597\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " Oct 30 13:55:01.571455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.571418 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-metrics-certs\") pod \"9a1973c8-eb1b-471e-a568-b81943e66597\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " Oct 30 13:55:01.571455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.571449 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1973c8-eb1b-471e-a568-b81943e66597-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1973c8-eb1b-471e-a568-b81943e66597-openshift-service-ca\") pod \"9a1973c8-eb1b-471e-a568-b81943e66597\" (UID: \"9a1973c8-eb1b-471e-a568-b81943e66597\") " Oct 30 13:55:01.571851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.571824 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1973c8-eb1b-471e-a568-b81943e66597-openshift-service-ca" pod "9a1973c8-eb1b-471e-a568-b81943e66597" (UID: "9a1973c8-eb1b-471e-a568-b81943e66597"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:01.572051 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.572036 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1973c8-eb1b-471e-a568-b81943e66597-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:01.573565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.573542 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1973c8-eb1b-471e-a568-b81943e66597-kube-api-access-czlvw" pod "9a1973c8-eb1b-471e-a568-b81943e66597" (UID: "9a1973c8-eb1b-471e-a568-b81943e66597"). InnerVolumeSpecName "kube-api-access-czlvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:01.573835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.573808 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-maas-api-tls" pod "9a1973c8-eb1b-471e-a568-b81943e66597" (UID: "9a1973c8-eb1b-471e-a568-b81943e66597"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:01.573835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.573823 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-metrics-certs" pod "9a1973c8-eb1b-471e-a568-b81943e66597" (UID: "9a1973c8-eb1b-471e-a568-b81943e66597"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:01.681290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.681206 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czlvw\" (UniqueName: \"kubernetes.io/projected/9a1973c8-eb1b-471e-a568-b81943e66597-kube-api-access-czlvw\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:01.681290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.681244 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:01.681290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.681257 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1973c8-eb1b-471e-a568-b81943e66597-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:01.851110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.851034 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:01.929886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.929854 5515 generic.go:356] "Generic (PLEG): container finished" podID="9a1973c8-eb1b-471e-a568-b81943e66597" containerID="53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8" exitCode=0 Oct 30 13:55:01.930082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.929900 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" Oct 30 13:55:01.930082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.929932 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" event={"ID":"9a1973c8-eb1b-471e-a568-b81943e66597","Type":"ContainerDied","Data":"53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8"} Oct 30 13:55:01.930082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.929991 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6" event={"ID":"9a1973c8-eb1b-471e-a568-b81943e66597","Type":"ContainerDied","Data":"933596e4ad9e00076c6da55c412e6d657f74b4760f20c82d95c79d78582286a3"} Oct 30 13:55:01.930082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.930019 5515 scope.go:122] "RemoveContainer" containerID="53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8" Oct 30 13:55:01.941145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.941126 5515 scope.go:122] "RemoveContainer" containerID="53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8" Oct 30 13:55:01.941426 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:01.941408 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8\": container with ID starting with 53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8 not found: ID does not exist" containerID="53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8" Oct 30 13:55:01.941482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.941433 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8"} err="failed to get container status \"53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8\": rpc error: code = NotFound desc = could not find container \"53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8\": container with ID starting with 53924a5cafb48ae9ef7d2b635114a8f4a903e5c79440483d1e749765a238bbd8 not found: ID does not exist" Oct 30 13:55:01.985700 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:01.985672 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6"] Oct 30 13:55:02.004815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:02.004782 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7a8b38-69869c95b4-m9hp6"] Oct 30 13:55:02.100867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:02.098154 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:02.531984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:02.531947 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:02.650334 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:02.650301 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph"] Oct 30 13:55:02.650575 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:02.650553 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" containerID="cri-o://3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74" gracePeriod=30 Oct 30 13:55:03.405622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.405587 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk"] Oct 30 13:55:03.405830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.405792 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" podUID="6b564df0-7119-482f-8404-9a78078481aa" containerName="maas-api" containerID="cri-o://57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9" gracePeriod=30 Oct 30 13:55:03.556812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.556776 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" podUID="ea2cbef4-f022-4196-afac-643eda0f7855" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:55:03.571423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.571370 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:03.694933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.694902 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:55:03.808898 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.808864 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b564df0-7119-482f-8404-9a78078481aa-kube-api-access-lsfmc\" (UniqueName: \"kubernetes.io/projected/6b564df0-7119-482f-8404-9a78078481aa-kube-api-access-lsfmc\") pod \"6b564df0-7119-482f-8404-9a78078481aa\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " Oct 30 13:55:03.809073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.808947 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-maas-api-tls\") pod \"6b564df0-7119-482f-8404-9a78078481aa\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " Oct 30 13:55:03.809073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.808985 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-metrics-certs\") pod \"6b564df0-7119-482f-8404-9a78078481aa\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " Oct 30 13:55:03.809073 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.809061 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b564df0-7119-482f-8404-9a78078481aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b564df0-7119-482f-8404-9a78078481aa-openshift-service-ca\") pod \"6b564df0-7119-482f-8404-9a78078481aa\" (UID: \"6b564df0-7119-482f-8404-9a78078481aa\") " Oct 30 13:55:03.809516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.809486 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b564df0-7119-482f-8404-9a78078481aa-openshift-service-ca" pod "6b564df0-7119-482f-8404-9a78078481aa" (UID: "6b564df0-7119-482f-8404-9a78078481aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:03.811175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.811152 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-maas-api-tls" pod "6b564df0-7119-482f-8404-9a78078481aa" (UID: "6b564df0-7119-482f-8404-9a78078481aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:03.811257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.811170 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b564df0-7119-482f-8404-9a78078481aa-kube-api-access-lsfmc" pod "6b564df0-7119-482f-8404-9a78078481aa" (UID: "6b564df0-7119-482f-8404-9a78078481aa"). InnerVolumeSpecName "kube-api-access-lsfmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:03.811316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.811298 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-metrics-certs" pod "6b564df0-7119-482f-8404-9a78078481aa" (UID: "6b564df0-7119-482f-8404-9a78078481aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:03.929578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.928397 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:03.929578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.928433 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b564df0-7119-482f-8404-9a78078481aa-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:03.929578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.928450 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsfmc\" (UniqueName: \"kubernetes.io/projected/6b564df0-7119-482f-8404-9a78078481aa-kube-api-access-lsfmc\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:03.929578 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.928464 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b564df0-7119-482f-8404-9a78078481aa-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:03.947967 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.947933 5515 generic.go:356] "Generic (PLEG): container finished" podID="6b564df0-7119-482f-8404-9a78078481aa" containerID="57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9" exitCode=0 Oct 30 13:55:03.948181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.947995 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" Oct 30 13:55:03.948181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.948025 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" event={"ID":"6b564df0-7119-482f-8404-9a78078481aa","Type":"ContainerDied","Data":"57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9"} Oct 30 13:55:03.948181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.948129 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk" event={"ID":"6b564df0-7119-482f-8404-9a78078481aa","Type":"ContainerDied","Data":"8e091e0831b2dd1d7f9021e360c2c0d94ddb2f7544d46d97040cd9c6e8044b37"} Oct 30 13:55:03.948181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.948161 5515 scope.go:122] "RemoveContainer" containerID="57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9" Oct 30 13:55:03.959244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.959223 5515 scope.go:122] "RemoveContainer" containerID="57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9" Oct 30 13:55:03.959579 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:03.959556 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9\": container with ID starting with 57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9 not found: ID does not exist" containerID="57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9" Oct 30 13:55:03.959654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:03.959592 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9"} err="failed to get container status \"57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9\": rpc error: code = NotFound desc = could not find container \"57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9\": container with ID starting with 57f67178b091abfa308da9e8d8ed21f07c11271e9e704f400c2b8881e8832ca9 not found: ID does not exist" Oct 30 13:55:04.003401 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:04.003370 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1973c8-eb1b-471e-a568-b81943e66597" path="/var/lib/kubelet/pods/9a1973c8-eb1b-471e-a568-b81943e66597/volumes" Oct 30 13:55:04.003775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:04.003757 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk"] Oct 30 13:55:04.006447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:04.006425 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1957dc-746c589cf6-vvxsk"] Oct 30 13:55:04.085227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:04.085201 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:04.265381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:04.265305 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:55:04.293298 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:04.293252 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:04.583150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:04.583067 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" podUID="fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:55:04.715289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:04.714620 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:05.023469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.022874 5515 generic.go:356] "Generic (PLEG): container finished" podID="fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" containerID="10c9716038d30923c0d8cfc46b1e964fb1c994ed7a823a3323e1a9242944479e" exitCode=0 Oct 30 13:55:05.023469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.023010 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" event={"ID":"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd","Type":"ContainerDied","Data":"10c9716038d30923c0d8cfc46b1e964fb1c994ed7a823a3323e1a9242944479e"} Oct 30 13:55:05.026984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.024699 5515 generic.go:356] "Generic (PLEG): container finished" podID="ea2cbef4-f022-4196-afac-643eda0f7855" containerID="ea78a51d9f07b96f4c3173e946a87d43bd5d8b59c5e543873b335fa5ddae7ded" exitCode=0 Oct 30 13:55:05.026984 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.024734 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" event={"ID":"ea2cbef4-f022-4196-afac-643eda0f7855","Type":"ContainerDied","Data":"ea78a51d9f07b96f4c3173e946a87d43bd5d8b59c5e543873b335fa5ddae7ded"} Oct 30 13:55:05.112332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.112304 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:55:05.142655 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.142606 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:55:05.200291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.198058 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2cbef4-f022-4196-afac-643eda0f7855-kube-api-access-lc4w8\" (UniqueName: \"kubernetes.io/projected/ea2cbef4-f022-4196-afac-643eda0f7855-kube-api-access-lc4w8\") pod \"ea2cbef4-f022-4196-afac-643eda0f7855\" (UID: \"ea2cbef4-f022-4196-afac-643eda0f7855\") " Oct 30 13:55:05.200291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.198128 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-kube-api-access-b9xms\" (UniqueName: \"kubernetes.io/projected/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-kube-api-access-b9xms\") pod \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\" (UID: \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\") " Oct 30 13:55:05.200291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.198318 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2cbef4-f022-4196-afac-643eda0f7855-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2cbef4-f022-4196-afac-643eda0f7855-plugins-config-volume\") pod \"ea2cbef4-f022-4196-afac-643eda0f7855\" (UID: \"ea2cbef4-f022-4196-afac-643eda0f7855\") " Oct 30 13:55:05.200291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.198416 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-plugins-config-volume\") pod \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\" (UID: \"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd\") " Oct 30 13:55:05.217795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.200963 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-plugins-config-volume" pod "fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" (UID: "fc85d4de-d18a-4213-bdb8-4d2f1c198fdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:05.217795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.204565 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2cbef4-f022-4196-afac-643eda0f7855-kube-api-access-lc4w8" pod "ea2cbef4-f022-4196-afac-643eda0f7855" (UID: "ea2cbef4-f022-4196-afac-643eda0f7855"). InnerVolumeSpecName "kube-api-access-lc4w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:05.217795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.205668 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2cbef4-f022-4196-afac-643eda0f7855-plugins-config-volume" pod "ea2cbef4-f022-4196-afac-643eda0f7855" (UID: "ea2cbef4-f022-4196-afac-643eda0f7855"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:05.217795 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.207369 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-kube-api-access-b9xms" pod "fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" (UID: "fc85d4de-d18a-4213-bdb8-4d2f1c198fdd"). InnerVolumeSpecName "kube-api-access-b9xms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:05.300224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.300179 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:05.300224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.300220 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc4w8\" (UniqueName: \"kubernetes.io/projected/ea2cbef4-f022-4196-afac-643eda0f7855-kube-api-access-lc4w8\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:05.300224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.300231 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xms\" (UniqueName: \"kubernetes.io/projected/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd-kube-api-access-b9xms\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:05.300224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:05.300240 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2cbef4-f022-4196-afac-643eda0f7855-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:06.017341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.005985 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b564df0-7119-482f-8404-9a78078481aa" path="/var/lib/kubelet/pods/6b564df0-7119-482f-8404-9a78078481aa/volumes" Oct 30 13:55:06.025505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.022067 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:06.036323 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.036259 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" Oct 30 13:55:06.036821 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.036800 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx" event={"ID":"fc85d4de-d18a-4213-bdb8-4d2f1c198fdd","Type":"ContainerDied","Data":"68bdb099800e89b39929f0320521ca75bde4d6e0b11343405e6f75e654ae2f0d"} Oct 30 13:55:06.036988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.036853 5515 scope.go:122] "RemoveContainer" containerID="10c9716038d30923c0d8cfc46b1e964fb1c994ed7a823a3323e1a9242944479e" Oct 30 13:55:06.038802 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.038742 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" event={"ID":"ea2cbef4-f022-4196-afac-643eda0f7855","Type":"ContainerDied","Data":"799cf04d0b8fbfd3e3da2c245e45487ad5d11e5095cb3a6e7471e9c09b505ecc"} Oct 30 13:55:06.038914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.038830 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk" Oct 30 13:55:06.048150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.048133 5515 scope.go:122] "RemoveContainer" containerID="ea78a51d9f07b96f4c3173e946a87d43bd5d8b59c5e543873b335fa5ddae7ded" Oct 30 13:55:06.121723 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.121690 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk"] Oct 30 13:55:06.124436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.124415 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7a8b38-8b9d9c555-h2shk"] Oct 30 13:55:06.158364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.158339 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx"] Oct 30 13:55:06.162319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:06.162300 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7a8b38-66dd77cbb-czsrx"] Oct 30 13:55:07.878535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:07.877837 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:08.007321 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.007292 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2cbef4-f022-4196-afac-643eda0f7855" path="/var/lib/kubelet/pods/ea2cbef4-f022-4196-afac-643eda0f7855/volumes" Oct 30 13:55:08.007797 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.007776 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" path="/var/lib/kubelet/pods/fc85d4de-d18a-4213-bdb8-4d2f1c198fdd/volumes" Oct 30 13:55:08.013521 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.013486 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:55:08.039720 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.039689 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-token\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-token\") pod \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " Oct 30 13:55:08.039868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.039736 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-envoy\") pod \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " Oct 30 13:55:08.039940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.039864 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-certs\") pod \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " Oct 30 13:55:08.039940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.039899 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-socket\") pod \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " Oct 30 13:55:08.039940 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.039936 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-credential-socket\") pod \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " Oct 30 13:55:08.040125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.039990 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istiod-ca-cert\") pod \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " Oct 30 13:55:08.040125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.040037 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-data\") pod \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " Oct 30 13:55:08.040241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.040136 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-podinfo\") pod \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " Oct 30 13:55:08.040241 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.040179 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-kube-api-access-mklgt\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-kube-api-access-mklgt\") pod \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\" (UID: \"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea\") " Oct 30 13:55:08.040478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.040452 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-credential-socket" pod "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" (UID: "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:08.040749 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.040721 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istiod-ca-cert" pod "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" (UID: "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:08.040926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.040900 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.040926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.040927 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.041098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.040992 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-certs" pod "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" (UID: "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:08.041098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.041064 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-socket" pod "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" (UID: "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:08.041622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.041598 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-data" pod "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" (UID: "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:08.043016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.042989 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-envoy" pod "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" (UID: "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:08.043661 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.043633 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-kube-api-access-mklgt" pod "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" (UID: "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea"). InnerVolumeSpecName "kube-api-access-mklgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:08.044693 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.044661 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-token" pod "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" (UID: "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:08.045460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.045435 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-podinfo" pod "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" (UID: "ed15e5d5-20e0-4eb1-8fd0-4905858cbaea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:55:08.050923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.050900 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:55:08.060606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.060578 5515 generic.go:356] "Generic (PLEG): container finished" podID="488a7efe-889c-4c6d-a9a1-5fee8a394212" containerID="5ff27a188a9b594686a406dba028f2c3cef939d51c878141133b28d57d3c3767" exitCode=0 Oct 30 13:55:08.060727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.060610 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" event={"ID":"488a7efe-889c-4c6d-a9a1-5fee8a394212","Type":"ContainerDied","Data":"5ff27a188a9b594686a406dba028f2c3cef939d51c878141133b28d57d3c3767"} Oct 30 13:55:08.062361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.062330 5515 generic.go:356] "Generic (PLEG): container finished" podID="fa629737-3309-43f6-8d6d-bea2f57918de" containerID="34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075" exitCode=0 Oct 30 13:55:08.062458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.062385 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" Oct 30 13:55:08.062458 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.062393 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" event={"ID":"fa629737-3309-43f6-8d6d-bea2f57918de","Type":"ContainerDied","Data":"34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075"} Oct 30 13:55:08.062597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.062459 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579" event={"ID":"fa629737-3309-43f6-8d6d-bea2f57918de","Type":"ContainerDied","Data":"4d60ff64c5ff65386c26c063fcdc32ce19382e37f19707fae5668de64446dad3"} Oct 30 13:55:08.062597 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.062490 5515 scope.go:122] "RemoveContainer" containerID="34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075" Oct 30 13:55:08.064807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.064783 5515 generic.go:356] "Generic (PLEG): container finished" podID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerID="3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74" exitCode=0 Oct 30 13:55:08.064923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.064826 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" Oct 30 13:55:08.064923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.064864 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" event={"ID":"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea","Type":"ContainerDied","Data":"3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74"} Oct 30 13:55:08.064923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.064907 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph" event={"ID":"ed15e5d5-20e0-4eb1-8fd0-4905858cbaea","Type":"ContainerDied","Data":"69dbeb65c6754d33da2535eb93922324d63387d65149ee4c6e64408636b3963e"} Oct 30 13:55:08.069738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.069719 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:55:08.074035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.074017 5515 scope.go:122] "RemoveContainer" containerID="34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075" Oct 30 13:55:08.074386 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:08.074363 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075\": container with ID starting with 34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075 not found: ID does not exist" containerID="34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075" Oct 30 13:55:08.074451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.074397 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075"} err="failed to get container status \"34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075\": rpc error: code = NotFound desc = could not find container \"34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075\": container with ID starting with 34bdd24384a9cf072eb3de2eeeb568dc43cf2c1f6f002278c829f475302bb075 not found: ID does not exist" Oct 30 13:55:08.074451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.074418 5515 scope.go:122] "RemoveContainer" containerID="3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74" Oct 30 13:55:08.094174 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.094152 5515 scope.go:122] "RemoveContainer" containerID="3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74" Oct 30 13:55:08.094530 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:08.094500 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74\": container with ID starting with 3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74 not found: ID does not exist" containerID="3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74" Oct 30 13:55:08.094615 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.094538 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74"} err="failed to get container status \"3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74\": rpc error: code = NotFound desc = could not find container \"3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74\": container with ID starting with 3316d178fc89b8eaa8f16cbb3430d5259b3e279541969ca9257ecf7c38a14d74 not found: ID does not exist" Oct 30 13:55:08.141584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.141555 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa629737-3309-43f6-8d6d-bea2f57918de-kube-api-access-lk4bf\" (UniqueName: \"kubernetes.io/projected/fa629737-3309-43f6-8d6d-bea2f57918de-kube-api-access-lk4bf\") pod \"fa629737-3309-43f6-8d6d-bea2f57918de\" (UID: \"fa629737-3309-43f6-8d6d-bea2f57918de\") " Oct 30 13:55:08.141732 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.141692 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488a7efe-889c-4c6d-a9a1-5fee8a394212-kube-api-access-kf5np\" (UniqueName: \"kubernetes.io/projected/488a7efe-889c-4c6d-a9a1-5fee8a394212-kube-api-access-kf5np\") pod \"488a7efe-889c-4c6d-a9a1-5fee8a394212\" (UID: \"488a7efe-889c-4c6d-a9a1-5fee8a394212\") " Oct 30 13:55:08.141828 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.141814 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa629737-3309-43f6-8d6d-bea2f57918de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa629737-3309-43f6-8d6d-bea2f57918de-plugins-config-volume\") pod \"fa629737-3309-43f6-8d6d-bea2f57918de\" (UID: \"fa629737-3309-43f6-8d6d-bea2f57918de\") " Oct 30 13:55:08.141911 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.141858 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488a7efe-889c-4c6d-a9a1-5fee8a394212-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488a7efe-889c-4c6d-a9a1-5fee8a394212-plugins-config-volume\") pod \"488a7efe-889c-4c6d-a9a1-5fee8a394212\" (UID: \"488a7efe-889c-4c6d-a9a1-5fee8a394212\") " Oct 30 13:55:08.142250 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.142223 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa629737-3309-43f6-8d6d-bea2f57918de-plugins-config-volume" pod "fa629737-3309-43f6-8d6d-bea2f57918de" (UID: "fa629737-3309-43f6-8d6d-bea2f57918de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:08.142361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.142311 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.142361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.142331 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mklgt\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-kube-api-access-mklgt\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.142361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.142344 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.142361 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.142359 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.142541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.142372 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.142541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.142386 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.142541 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.142401 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.142748 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.142724 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488a7efe-889c-4c6d-a9a1-5fee8a394212-plugins-config-volume" pod "488a7efe-889c-4c6d-a9a1-5fee8a394212" (UID: "488a7efe-889c-4c6d-a9a1-5fee8a394212"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:08.143703 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.143682 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa629737-3309-43f6-8d6d-bea2f57918de-kube-api-access-lk4bf" pod "fa629737-3309-43f6-8d6d-bea2f57918de" (UID: "fa629737-3309-43f6-8d6d-bea2f57918de"). InnerVolumeSpecName "kube-api-access-lk4bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:08.143900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.143882 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488a7efe-889c-4c6d-a9a1-5fee8a394212-kube-api-access-kf5np" pod "488a7efe-889c-4c6d-a9a1-5fee8a394212" (UID: "488a7efe-889c-4c6d-a9a1-5fee8a394212"). InnerVolumeSpecName "kube-api-access-kf5np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:08.168462 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.168436 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph"] Oct 30 13:55:08.173120 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.173080 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vkjph"] Oct 30 13:55:08.243770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.243694 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf5np\" (UniqueName: \"kubernetes.io/projected/488a7efe-889c-4c6d-a9a1-5fee8a394212-kube-api-access-kf5np\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.243770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.243728 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa629737-3309-43f6-8d6d-bea2f57918de-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.243770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.243744 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488a7efe-889c-4c6d-a9a1-5fee8a394212-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.243770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.243758 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4bf\" (UniqueName: \"kubernetes.io/projected/fa629737-3309-43f6-8d6d-bea2f57918de-kube-api-access-lk4bf\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:08.409013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.408979 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579"] Oct 30 13:55:08.416021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.415991 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8740d4-59db5fb696-7b579"] Oct 30 13:55:08.632141 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.632081 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:08.833125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.832518 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2"] Oct 30 13:55:08.833125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.832760 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" containerID="cri-o://963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7" gracePeriod=30 Oct 30 13:55:08.841605 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:08.841577 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:09.072135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:09.072050 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" Oct 30 13:55:09.072613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:09.072049 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5" event={"ID":"488a7efe-889c-4c6d-a9a1-5fee8a394212","Type":"ContainerDied","Data":"dbaa4535f35afd39dd6b1f741e23f937f775681d37a865c686b0cdacd88e40d4"} Oct 30 13:55:09.072613 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:09.072196 5515 scope.go:122] "RemoveContainer" containerID="5ff27a188a9b594686a406dba028f2c3cef939d51c878141133b28d57d3c3767" Oct 30 13:55:09.123641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:09.123614 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:09.124282 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:09.124247 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5"] Oct 30 13:55:09.129178 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:09.129157 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8740d4-675f87d779-szvh5"] Oct 30 13:55:09.608722 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:09.608683 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:10.002353 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:10.002325 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488a7efe-889c-4c6d-a9a1-5fee8a394212" path="/var/lib/kubelet/pods/488a7efe-889c-4c6d-a9a1-5fee8a394212/volumes" Oct 30 13:55:10.002862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:10.002845 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" path="/var/lib/kubelet/pods/ed15e5d5-20e0-4eb1-8fd0-4905858cbaea/volumes" Oct 30 13:55:10.003262 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:10.003246 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa629737-3309-43f6-8d6d-bea2f57918de" path="/var/lib/kubelet/pods/fa629737-3309-43f6-8d6d-bea2f57918de/volumes" Oct 30 13:55:10.642890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:10.642861 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:10.863853 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:10.863816 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:11.083704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:11.083629 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:11.521487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:11.521451 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:12.392938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:12.392909 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:12.603046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:12.603012 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:12.974122 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:12.974082 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:13.197518 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:13.197478 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:13.405633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:13.405607 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:14.103667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.103613 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:55:14.107924 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.107893 5515 generic.go:356] "Generic (PLEG): container finished" podID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerID="963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7" exitCode=0 Oct 30 13:55:14.108064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.107934 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" Oct 30 13:55:14.108064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.107970 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" event={"ID":"d6f09afe-2478-4f1c-b9c0-dca74dc14765","Type":"ContainerDied","Data":"963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7"} Oct 30 13:55:14.108064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.108029 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2" event={"ID":"d6f09afe-2478-4f1c-b9c0-dca74dc14765","Type":"ContainerDied","Data":"886cab76b634ae7f517469edcdafd454cc6cfc848590b059640e420c77b49516"} Oct 30 13:55:14.108183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.108068 5515 scope.go:122] "RemoveContainer" containerID="963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7" Oct 30 13:55:14.117439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.117423 5515 scope.go:122] "RemoveContainer" containerID="a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d" Oct 30 13:55:14.126472 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.126193 5515 scope.go:122] "RemoveContainer" containerID="963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7" Oct 30 13:55:14.126602 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:14.126468 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7\": container with ID starting with 963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7 not found: ID does not exist" containerID="963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7" Oct 30 13:55:14.126602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.126492 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7"} err="failed to get container status \"963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7\": rpc error: code = NotFound desc = could not find container \"963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7\": container with ID starting with 963b95e900ca3a5809e7c30dfc888d839da3038862706f5bbb8014089aaf4fd7 not found: ID does not exist" Oct 30 13:55:14.126602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.126509 5515 scope.go:122] "RemoveContainer" containerID="a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d" Oct 30 13:55:14.126794 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:14.126776 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d\": container with ID starting with a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d not found: ID does not exist" containerID="a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d" Oct 30 13:55:14.126834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.126803 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d"} err="failed to get container status \"a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d\": rpc error: code = NotFound desc = could not find container \"a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d\": container with ID starting with a52ef891b9e44d05af26c6e36105918eb980262d17c356cffc7fc3827d7f0a7d not found: ID does not exist" Oct 30 13:55:14.246311 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246261 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-credential-socket\") pod \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " Oct 30 13:55:14.246477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246326 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-data\") pod \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " Oct 30 13:55:14.246477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246363 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-certs\") pod \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " Oct 30 13:55:14.246477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246383 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-socket\") pod \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " Oct 30 13:55:14.246477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246434 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-envoy\") pod \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " Oct 30 13:55:14.246656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246578 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-podinfo\") pod \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " Oct 30 13:55:14.246656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246585 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-credential-socket" pod "d6f09afe-2478-4f1c-b9c0-dca74dc14765" (UID: "d6f09afe-2478-4f1c-b9c0-dca74dc14765"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:14.246656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246593 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-data" pod "d6f09afe-2478-4f1c-b9c0-dca74dc14765" (UID: "d6f09afe-2478-4f1c-b9c0-dca74dc14765"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:14.246656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246620 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-token\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-token\") pod \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " Oct 30 13:55:14.246836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246700 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-socket" pod "d6f09afe-2478-4f1c-b9c0-dca74dc14765" (UID: "d6f09afe-2478-4f1c-b9c0-dca74dc14765"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:14.246836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246710 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-certs" pod "d6f09afe-2478-4f1c-b9c0-dca74dc14765" (UID: "d6f09afe-2478-4f1c-b9c0-dca74dc14765"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:14.246836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246737 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istiod-ca-cert\") pod \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " Oct 30 13:55:14.246836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.246775 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-kube-api-access-vp7b6\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-kube-api-access-vp7b6\") pod \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\" (UID: \"d6f09afe-2478-4f1c-b9c0-dca74dc14765\") " Oct 30 13:55:14.247071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.247039 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istiod-ca-cert" pod "d6f09afe-2478-4f1c-b9c0-dca74dc14765" (UID: "d6f09afe-2478-4f1c-b9c0-dca74dc14765"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:14.247346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.247312 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:14.247346 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.247346 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:14.247524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.247356 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:14.247524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.247370 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:14.247524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.247383 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:14.248963 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.248934 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-podinfo" pod "d6f09afe-2478-4f1c-b9c0-dca74dc14765" (UID: "d6f09afe-2478-4f1c-b9c0-dca74dc14765"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:55:14.249080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.248967 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-kube-api-access-vp7b6" pod "d6f09afe-2478-4f1c-b9c0-dca74dc14765" (UID: "d6f09afe-2478-4f1c-b9c0-dca74dc14765"). InnerVolumeSpecName "kube-api-access-vp7b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:14.249080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.249075 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-envoy" pod "d6f09afe-2478-4f1c-b9c0-dca74dc14765" (UID: "d6f09afe-2478-4f1c-b9c0-dca74dc14765"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:14.249164 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.249117 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-token" pod "d6f09afe-2478-4f1c-b9c0-dca74dc14765" (UID: "d6f09afe-2478-4f1c-b9c0-dca74dc14765"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:14.348354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.348328 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7b6\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-kube-api-access-vp7b6\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:14.348354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.348350 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:14.348354 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.348360 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:14.348542 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.348369 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f09afe-2478-4f1c-b9c0-dca74dc14765-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:14.458746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.458718 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2"] Oct 30 13:55:14.470634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:14.470611 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8740d4-openshift-default-79ccb5884f-cllf2"] Oct 30 13:55:15.195734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:15.195685 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:16.001856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:16.001824 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" path="/var/lib/kubelet/pods/d6f09afe-2478-4f1c-b9c0-dca74dc14765/volumes" Oct 30 13:55:17.008247 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:17.008214 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:17.269668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:17.269553 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:18.593514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:18.593483 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:19.968678 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:19.968641 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w"] Oct 30 13:55:20.196621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:20.196574 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:20.428189 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:20.428153 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:21.908688 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:21.908650 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:21.925946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:21.925912 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq"] Oct 30 13:55:22.182730 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:22.182639 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:22.434598 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:22.434519 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:23.372870 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:23.372839 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:23.595834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:23.595804 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:24.294039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:24.294008 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck"] Oct 30 13:55:24.529753 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:24.529717 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:24.781380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:24.781351 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:24.968903 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:24.968868 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" podUID="60e64645-cc47-496b-980b-f22d6ed896e7" containerName="payload-pre-processing" containerID="cri-o://36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc" gracePeriod=25 Oct 30 13:55:25.292535 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:25.292495 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:26.926453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:26.926389 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" podUID="ca2ea174-b297-44ab-96ea-ee1eaf59dbde" containerName="payload-processing" containerID="cri-o://6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707" gracePeriod=25 Oct 30 13:55:26.995914 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:26.995865 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:28.635461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:28.635424 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:28.987552 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:28.987463 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:55:29.294465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:29.294383 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" podUID="d8b1762a-e274-4dad-9f2c-a2c5f6e58380" containerName="payload-pre-processing" containerID="cri-o://b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14" gracePeriod=25 Oct 30 13:55:30.257263 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:30.257228 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:31.929993 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:31.929949 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" podUID="d8b1762a-e274-4dad-9f2c-a2c5f6e58380" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:55:31.984314 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:31.978909 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:32.186627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:32.186545 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:33.199441 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.199405 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq"] Oct 30 13:55:33.199939 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.199684 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" podUID="080d1adc-43f1-4344-97a4-618a8b5ea618" containerName="maas-api" containerID="cri-o://abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08" gracePeriod=30 Oct 30 13:55:33.369647 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.369614 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:33.496454 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.496428 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:55:33.575020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.574977 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080d1adc-43f1-4344-97a4-618a8b5ea618-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d1adc-43f1-4344-97a4-618a8b5ea618-openshift-service-ca\") pod \"080d1adc-43f1-4344-97a4-618a8b5ea618\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " Oct 30 13:55:33.575229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.575092 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080d1adc-43f1-4344-97a4-618a8b5ea618-kube-api-access-575lb\" (UniqueName: \"kubernetes.io/projected/080d1adc-43f1-4344-97a4-618a8b5ea618-kube-api-access-575lb\") pod \"080d1adc-43f1-4344-97a4-618a8b5ea618\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " Oct 30 13:55:33.575229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.575138 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-maas-api-tls\") pod \"080d1adc-43f1-4344-97a4-618a8b5ea618\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " Oct 30 13:55:33.575229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.575175 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-metrics-certs\") pod \"080d1adc-43f1-4344-97a4-618a8b5ea618\" (UID: \"080d1adc-43f1-4344-97a4-618a8b5ea618\") " Oct 30 13:55:33.575517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.575431 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080d1adc-43f1-4344-97a4-618a8b5ea618-openshift-service-ca" pod "080d1adc-43f1-4344-97a4-618a8b5ea618" (UID: "080d1adc-43f1-4344-97a4-618a8b5ea618"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:33.575729 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.575708 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d1adc-43f1-4344-97a4-618a8b5ea618-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:33.577752 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.577724 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-metrics-certs" pod "080d1adc-43f1-4344-97a4-618a8b5ea618" (UID: "080d1adc-43f1-4344-97a4-618a8b5ea618"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:33.577900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.577815 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-maas-api-tls" pod "080d1adc-43f1-4344-97a4-618a8b5ea618" (UID: "080d1adc-43f1-4344-97a4-618a8b5ea618"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:33.578231 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.578201 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080d1adc-43f1-4344-97a4-618a8b5ea618-kube-api-access-575lb" pod "080d1adc-43f1-4344-97a4-618a8b5ea618" (UID: "080d1adc-43f1-4344-97a4-618a8b5ea618"). InnerVolumeSpecName "kube-api-access-575lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:33.677359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.677320 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-575lb\" (UniqueName: \"kubernetes.io/projected/080d1adc-43f1-4344-97a4-618a8b5ea618-kube-api-access-575lb\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:33.677359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.677357 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:33.677622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.677371 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d1adc-43f1-4344-97a4-618a8b5ea618-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:33.835788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:33.835661 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:34.016451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.016420 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" podUID="60e64645-cc47-496b-980b-f22d6ed896e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:55:34.039378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.039317 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:34.246566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.246529 5515 generic.go:356] "Generic (PLEG): container finished" podID="080d1adc-43f1-4344-97a4-618a8b5ea618" containerID="abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08" exitCode=0 Oct 30 13:55:34.246986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.246579 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" Oct 30 13:55:34.246986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.246614 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" event={"ID":"080d1adc-43f1-4344-97a4-618a8b5ea618","Type":"ContainerDied","Data":"abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08"} Oct 30 13:55:34.246986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.246672 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq" event={"ID":"080d1adc-43f1-4344-97a4-618a8b5ea618","Type":"ContainerDied","Data":"16a5ee2232a238bcea8f5d71235b2932e421fa8cf1961c46d07993742d2d87b0"} Oct 30 13:55:34.246986 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.246695 5515 scope.go:122] "RemoveContainer" containerID="abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08" Oct 30 13:55:34.256332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.256315 5515 scope.go:122] "RemoveContainer" containerID="abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08" Oct 30 13:55:34.256589 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:34.256571 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08\": container with ID starting with abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08 not found: ID does not exist" containerID="abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08" Oct 30 13:55:34.256668 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.256598 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08"} err="failed to get container status \"abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08\": rpc error: code = NotFound desc = could not find container \"abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08\": container with ID starting with abb7f6080ec3511106026991afd63d1bf80067139d77c706ffbfe0a692a98a08 not found: ID does not exist" Oct 30 13:55:34.298341 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.298311 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq"] Oct 30 13:55:34.310517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.310493 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-98dabb-5db8946d74-ns7jq"] Oct 30 13:55:34.500337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:34.500243 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:35.846825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:35.846776 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:36.001138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:36.001099 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080d1adc-43f1-4344-97a4-618a8b5ea618" path="/var/lib/kubelet/pods/080d1adc-43f1-4344-97a4-618a8b5ea618/volumes" Oct 30 13:55:36.475165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:36.475127 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" podUID="ca2ea174-b297-44ab-96ea-ee1eaf59dbde" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:55:37.481934 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:37.481899 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:38.766690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:38.766654 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s"] Oct 30 13:55:38.767337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:38.766954 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" podUID="ff901dc4-d932-4a95-8e36-e1ccc2ab4050" containerName="maas-api" containerID="cri-o://d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d" gracePeriod=30 Oct 30 13:55:38.821998 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:38.821961 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:39.025132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.025098 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:39.039283 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.039249 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:55:39.142498 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.142461 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-maas-api-tls\") pod \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " Oct 30 13:55:39.142708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.142552 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-openshift-service-ca\") pod \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " Oct 30 13:55:39.142708 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.142664 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-metrics-certs\") pod \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " Oct 30 13:55:39.142838 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.142771 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-kube-api-access-78grm\" (UniqueName: \"kubernetes.io/projected/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-kube-api-access-78grm\") pod \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\" (UID: \"ff901dc4-d932-4a95-8e36-e1ccc2ab4050\") " Oct 30 13:55:39.142933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.142910 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-openshift-service-ca" pod "ff901dc4-d932-4a95-8e36-e1ccc2ab4050" (UID: "ff901dc4-d932-4a95-8e36-e1ccc2ab4050"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:39.143836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.143804 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:39.145046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.145024 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-metrics-certs" pod "ff901dc4-d932-4a95-8e36-e1ccc2ab4050" (UID: "ff901dc4-d932-4a95-8e36-e1ccc2ab4050"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:39.147246 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.147225 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-maas-api-tls" pod "ff901dc4-d932-4a95-8e36-e1ccc2ab4050" (UID: "ff901dc4-d932-4a95-8e36-e1ccc2ab4050"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:39.147436 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.147416 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-kube-api-access-78grm" pod "ff901dc4-d932-4a95-8e36-e1ccc2ab4050" (UID: "ff901dc4-d932-4a95-8e36-e1ccc2ab4050"). InnerVolumeSpecName "kube-api-access-78grm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:39.244954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.244917 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:39.244954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.244948 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78grm\" (UniqueName: \"kubernetes.io/projected/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-kube-api-access-78grm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:39.244954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.244960 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff901dc4-d932-4a95-8e36-e1ccc2ab4050-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:39.279805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.279716 5515 generic.go:356] "Generic (PLEG): container finished" podID="ff901dc4-d932-4a95-8e36-e1ccc2ab4050" containerID="d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d" exitCode=0 Oct 30 13:55:39.279805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.279763 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" Oct 30 13:55:39.280061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.279809 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" event={"ID":"ff901dc4-d932-4a95-8e36-e1ccc2ab4050","Type":"ContainerDied","Data":"d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d"} Oct 30 13:55:39.280061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.279868 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s" event={"ID":"ff901dc4-d932-4a95-8e36-e1ccc2ab4050","Type":"ContainerDied","Data":"56044dd85482fe0258474fcad44ef0fbeb80a21fc7ad450f097717a088c641ee"} Oct 30 13:55:39.280061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.279895 5515 scope.go:122] "RemoveContainer" containerID="d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d" Oct 30 13:55:39.294570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.294541 5515 scope.go:122] "RemoveContainer" containerID="d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d" Oct 30 13:55:39.294927 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:39.294899 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d\": container with ID starting with d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d not found: ID does not exist" containerID="d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d" Oct 30 13:55:39.295019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.294938 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d"} err="failed to get container status \"d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d\": rpc error: code = NotFound desc = could not find container \"d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d\": container with ID starting with d927b37b7d2337ad6a38bdd415d4a4d60b12c27b9aca941f4ce8885b3d8e7f4d not found: ID does not exist" Oct 30 13:55:39.333834 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.333792 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s"] Oct 30 13:55:39.337805 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.337776 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d2a0b-56d495c765-v9q8s"] Oct 30 13:55:39.343146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.343124 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:39.600446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.600370 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:39.961837 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:39.961808 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:40.003445 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:40.003403 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff901dc4-d932-4a95-8e36-e1ccc2ab4050" path="/var/lib/kubelet/pods/ff901dc4-d932-4a95-8e36-e1ccc2ab4050/volumes" Oct 30 13:55:40.173971 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:40.173939 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:40.596379 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:40.596349 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:41.499568 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:41.499541 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:41.721202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:41.721173 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:41.930221 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:41.930181 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" podUID="d8b1762a-e274-4dad-9f2c-a2c5f6e58380" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:55:41.992659 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:41.989402 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:42.201041 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:42.200941 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:42.412516 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:42.412480 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:43.760016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:43.759983 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:44.016076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:44.015979 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" podUID="60e64645-cc47-496b-980b-f22d6ed896e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:55:45.064175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.064142 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:55:45.225818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.225751 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:55:45.318685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.318647 5515 generic.go:356] "Generic (PLEG): container finished" podID="60e64645-cc47-496b-980b-f22d6ed896e7" containerID="36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc" exitCode=0 Oct 30 13:55:45.318858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.318694 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" Oct 30 13:55:45.318858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.318735 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" event={"ID":"60e64645-cc47-496b-980b-f22d6ed896e7","Type":"ContainerDied","Data":"36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc"} Oct 30 13:55:45.318858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.318786 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w" event={"ID":"60e64645-cc47-496b-980b-f22d6ed896e7","Type":"ContainerDied","Data":"26afd6712b533c57963b4ad2f3b019b40f1cd90afed932485078cac3c485e58b"} Oct 30 13:55:45.318858 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.318809 5515 scope.go:122] "RemoveContainer" containerID="36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc" Oct 30 13:55:45.325501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.325474 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e64645-cc47-496b-980b-f22d6ed896e7-kube-api-access-qtmkj\" (UniqueName: \"kubernetes.io/projected/60e64645-cc47-496b-980b-f22d6ed896e7-kube-api-access-qtmkj\") pod \"60e64645-cc47-496b-980b-f22d6ed896e7\" (UID: \"60e64645-cc47-496b-980b-f22d6ed896e7\") " Oct 30 13:55:45.325656 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.325603 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e64645-cc47-496b-980b-f22d6ed896e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e64645-cc47-496b-980b-f22d6ed896e7-plugins-config-volume\") pod \"60e64645-cc47-496b-980b-f22d6ed896e7\" (UID: \"60e64645-cc47-496b-980b-f22d6ed896e7\") " Oct 30 13:55:45.326054 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.326029 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e64645-cc47-496b-980b-f22d6ed896e7-plugins-config-volume" pod "60e64645-cc47-496b-980b-f22d6ed896e7" (UID: "60e64645-cc47-496b-980b-f22d6ed896e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:45.327623 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.327599 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e64645-cc47-496b-980b-f22d6ed896e7-kube-api-access-qtmkj" pod "60e64645-cc47-496b-980b-f22d6ed896e7" (UID: "60e64645-cc47-496b-980b-f22d6ed896e7"). InnerVolumeSpecName "kube-api-access-qtmkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:45.328780 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.328755 5515 scope.go:122] "RemoveContainer" containerID="36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc" Oct 30 13:55:45.329096 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:45.329069 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc\": container with ID starting with 36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc not found: ID does not exist" containerID="36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc" Oct 30 13:55:45.329195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.329107 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc"} err="failed to get container status \"36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc\": rpc error: code = NotFound desc = could not find container \"36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc\": container with ID starting with 36142d262aa475325fe3cb0abe786a273eea4c7f71a7d8712950345b97a589bc not found: ID does not exist" Oct 30 13:55:45.426724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.426687 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtmkj\" (UniqueName: \"kubernetes.io/projected/60e64645-cc47-496b-980b-f22d6ed896e7-kube-api-access-qtmkj\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:45.426724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.426715 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e64645-cc47-496b-980b-f22d6ed896e7-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:45.509306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.509190 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:45.671726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.671693 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w"] Oct 30 13:55:45.675482 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:45.675447 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e9cb4a-68c7dd7ff-xxh4w"] Oct 30 13:55:46.001620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:46.001589 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e64645-cc47-496b-980b-f22d6ed896e7" path="/var/lib/kubelet/pods/60e64645-cc47-496b-980b-f22d6ed896e7/volumes" Oct 30 13:55:46.475625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:46.475591 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" podUID="ca2ea174-b297-44ab-96ea-ee1eaf59dbde" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:55:47.151160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.151125 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:47.191887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.191825 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:55:47.254431 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.254395 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-kube-api-access-nn8cl\" (UniqueName: \"kubernetes.io/projected/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-kube-api-access-nn8cl\") pod \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\" (UID: \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\") " Oct 30 13:55:47.254631 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.254481 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-plugins-config-volume\") pod \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\" (UID: \"ca2ea174-b297-44ab-96ea-ee1eaf59dbde\") " Oct 30 13:55:47.255072 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.254967 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-plugins-config-volume" pod "ca2ea174-b297-44ab-96ea-ee1eaf59dbde" (UID: "ca2ea174-b297-44ab-96ea-ee1eaf59dbde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:47.257209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.257168 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-kube-api-access-nn8cl" pod "ca2ea174-b297-44ab-96ea-ee1eaf59dbde" (UID: "ca2ea174-b297-44ab-96ea-ee1eaf59dbde"). InnerVolumeSpecName "kube-api-access-nn8cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:47.333380 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.333347 5515 generic.go:356] "Generic (PLEG): container finished" podID="ca2ea174-b297-44ab-96ea-ee1eaf59dbde" containerID="6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707" exitCode=0 Oct 30 13:55:47.333565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.333392 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" Oct 30 13:55:47.333565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.333436 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" event={"ID":"ca2ea174-b297-44ab-96ea-ee1eaf59dbde","Type":"ContainerDied","Data":"6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707"} Oct 30 13:55:47.333565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.333490 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq" event={"ID":"ca2ea174-b297-44ab-96ea-ee1eaf59dbde","Type":"ContainerDied","Data":"25566b277bab95c867b0c4aef2d24403e73d617adf412823b54024b239593b64"} Oct 30 13:55:47.333565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.333512 5515 scope.go:122] "RemoveContainer" containerID="6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707" Oct 30 13:55:47.344745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.344721 5515 scope.go:122] "RemoveContainer" containerID="6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707" Oct 30 13:55:47.345072 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:47.345045 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707\": container with ID starting with 6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707 not found: ID does not exist" containerID="6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707" Oct 30 13:55:47.345139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.345083 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707"} err="failed to get container status \"6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707\": rpc error: code = NotFound desc = could not find container \"6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707\": container with ID starting with 6ec7e055b9fb5f260e57c9c40174566b4ce176c6179f6a95d781db95d4462707 not found: ID does not exist" Oct 30 13:55:47.356537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.356511 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:47.356537 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.356535 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn8cl\" (UniqueName: \"kubernetes.io/projected/ca2ea174-b297-44ab-96ea-ee1eaf59dbde-kube-api-access-nn8cl\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:47.393625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.393587 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq"] Oct 30 13:55:47.406245 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.406217 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-98dabb-67d7c4fcd-87vkq"] Oct 30 13:55:47.538061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:47.537977 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:48.008868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:48.008833 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2ea174-b297-44ab-96ea-ee1eaf59dbde" path="/var/lib/kubelet/pods/ca2ea174-b297-44ab-96ea-ee1eaf59dbde/volumes" Oct 30 13:55:48.584091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:48.584046 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:48.796626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:48.796592 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:48.940071 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:48.940031 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt"] Oct 30 13:55:48.940439 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:48.940413 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" containerID="cri-o://6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b" gracePeriod=30 Oct 30 13:55:49.005280 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:49.005240 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:49.238157 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:49.238066 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:49.562532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:49.562508 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:55:49.585634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:49.585607 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-kube-api-access-q5wnn\" (UniqueName: \"kubernetes.io/projected/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-kube-api-access-q5wnn\") pod \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\" (UID: \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\") " Oct 30 13:55:49.586040 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:49.585757 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-plugins-config-volume\") pod \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\" (UID: \"d8b1762a-e274-4dad-9f2c-a2c5f6e58380\") " Oct 30 13:55:49.586230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:49.586197 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-plugins-config-volume" pod "d8b1762a-e274-4dad-9f2c-a2c5f6e58380" (UID: "d8b1762a-e274-4dad-9f2c-a2c5f6e58380"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:49.587897 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:49.587864 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-kube-api-access-q5wnn" pod "d8b1762a-e274-4dad-9f2c-a2c5f6e58380" (UID: "d8b1762a-e274-4dad-9f2c-a2c5f6e58380"). InnerVolumeSpecName "kube-api-access-q5wnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:49.687601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:49.687566 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wnn\" (UniqueName: \"kubernetes.io/projected/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-kube-api-access-q5wnn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:49.687601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:49.687599 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b1762a-e274-4dad-9f2c-a2c5f6e58380-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:50.108822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.108781 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:50.322045 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.321956 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:50.355097 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.355066 5515 generic.go:356] "Generic (PLEG): container finished" podID="d8b1762a-e274-4dad-9f2c-a2c5f6e58380" containerID="b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14" exitCode=0 Oct 30 13:55:50.355284 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.355113 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" Oct 30 13:55:50.355284 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.355146 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" event={"ID":"d8b1762a-e274-4dad-9f2c-a2c5f6e58380","Type":"ContainerDied","Data":"b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14"} Oct 30 13:55:50.355284 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.355206 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck" event={"ID":"d8b1762a-e274-4dad-9f2c-a2c5f6e58380","Type":"ContainerDied","Data":"419f9940f658270c2f32a5b57c8ef026e6123108abde26311a452ff1eae3961e"} Oct 30 13:55:50.355284 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.355234 5515 scope.go:122] "RemoveContainer" containerID="b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14" Oct 30 13:55:50.365728 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.365707 5515 scope.go:122] "RemoveContainer" containerID="b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14" Oct 30 13:55:50.366001 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:50.365977 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14\": container with ID starting with b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14 not found: ID does not exist" containerID="b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14" Oct 30 13:55:50.366061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.366009 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14"} err="failed to get container status \"b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14\": rpc error: code = NotFound desc = could not find container \"b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14\": container with ID starting with b3dd47cba6e1302575a62574f6ea0b8cc3338b4ad0d62909e0ed430af1ebcc14 not found: ID does not exist" Oct 30 13:55:50.406830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.406793 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck"] Oct 30 13:55:50.419213 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.419180 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2d2a0b-55c8d7f867-wlpck"] Oct 30 13:55:50.707566 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:50.707532 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:52.003083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:52.003039 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b1762a-e274-4dad-9f2c-a2c5f6e58380" path="/var/lib/kubelet/pods/d8b1762a-e274-4dad-9f2c-a2c5f6e58380/volumes" Oct 30 13:55:52.504825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:52.504793 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:53.340836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:53.340804 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:53.554306 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:53.554255 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:54.226695 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.226667 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:55:54.292166 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.292121 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:54.343579 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.343546 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-podinfo\") pod \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " Oct 30 13:55:54.344104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.343597 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-data\") pod \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " Oct 30 13:55:54.344104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.343621 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istiod-ca-cert\") pod \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " Oct 30 13:55:54.344104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.343699 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-envoy\") pod \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " Oct 30 13:55:54.344104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.343857 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-kube-api-access-rkw4h\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-kube-api-access-rkw4h\") pod \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " Oct 30 13:55:54.344104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.343884 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-socket\") pod \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " Oct 30 13:55:54.344104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.343925 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-certs\") pod \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " Oct 30 13:55:54.344104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.343952 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-data" pod "04d0d438-e1e1-434a-9b86-abf3f11e7f2b" (UID: "04d0d438-e1e1-434a-9b86-abf3f11e7f2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:54.344104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.343978 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-credential-socket\") pod \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " Oct 30 13:55:54.344104 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.344039 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-token\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-token\") pod \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\" (UID: \"04d0d438-e1e1-434a-9b86-abf3f11e7f2b\") " Oct 30 13:55:54.344650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.344179 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istiod-ca-cert" pod "04d0d438-e1e1-434a-9b86-abf3f11e7f2b" (UID: "04d0d438-e1e1-434a-9b86-abf3f11e7f2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:54.344650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.344199 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-socket" pod "04d0d438-e1e1-434a-9b86-abf3f11e7f2b" (UID: "04d0d438-e1e1-434a-9b86-abf3f11e7f2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:54.344650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.344416 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-certs" pod "04d0d438-e1e1-434a-9b86-abf3f11e7f2b" (UID: "04d0d438-e1e1-434a-9b86-abf3f11e7f2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:54.344650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.344430 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-credential-socket" pod "04d0d438-e1e1-434a-9b86-abf3f11e7f2b" (UID: "04d0d438-e1e1-434a-9b86-abf3f11e7f2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:54.344909 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.344879 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:54.345115 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.345095 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:54.345193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.345122 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:54.345193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.345138 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:54.345193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.345153 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:54.346519 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.346467 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-envoy" pod "04d0d438-e1e1-434a-9b86-abf3f11e7f2b" (UID: "04d0d438-e1e1-434a-9b86-abf3f11e7f2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:55:54.346622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.346565 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-podinfo" pod "04d0d438-e1e1-434a-9b86-abf3f11e7f2b" (UID: "04d0d438-e1e1-434a-9b86-abf3f11e7f2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:55:54.346622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.346600 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-token" pod "04d0d438-e1e1-434a-9b86-abf3f11e7f2b" (UID: "04d0d438-e1e1-434a-9b86-abf3f11e7f2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:54.346887 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.346870 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-kube-api-access-rkw4h" pod "04d0d438-e1e1-434a-9b86-abf3f11e7f2b" (UID: "04d0d438-e1e1-434a-9b86-abf3f11e7f2b"). InnerVolumeSpecName "kube-api-access-rkw4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:54.381634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.381596 5515 generic.go:356] "Generic (PLEG): container finished" podID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerID="6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b" exitCode=0 Oct 30 13:55:54.381808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.381646 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" Oct 30 13:55:54.381808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.381678 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" event={"ID":"04d0d438-e1e1-434a-9b86-abf3f11e7f2b","Type":"ContainerDied","Data":"6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b"} Oct 30 13:55:54.381808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.381734 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt" event={"ID":"04d0d438-e1e1-434a-9b86-abf3f11e7f2b","Type":"ContainerDied","Data":"196fa4becb859185b464f52848771127f36e99095b1257846d5ef3822e7395e9"} Oct 30 13:55:54.381808 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.381757 5515 scope.go:122] "RemoveContainer" containerID="6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b" Oct 30 13:55:54.392483 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.392463 5515 scope.go:122] "RemoveContainer" containerID="6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b" Oct 30 13:55:54.392761 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:54.392742 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b\": container with ID starting with 6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b not found: ID does not exist" containerID="6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b" Oct 30 13:55:54.392828 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.392772 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b"} err="failed to get container status \"6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b\": rpc error: code = NotFound desc = could not find container \"6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b\": container with ID starting with 6a83bc077cc4dec9a78ff0a8da5547526ac8ad6513792d2d0d043e629b003e3b not found: ID does not exist" Oct 30 13:55:54.434509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.434465 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt"] Oct 30 13:55:54.438177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.438145 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-4dwgt"] Oct 30 13:55:54.446701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.446672 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkw4h\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-kube-api-access-rkw4h\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:54.446701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.446699 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:54.446874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.446709 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:54.446874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:54.446717 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d0d438-e1e1-434a-9b86-abf3f11e7f2b-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:55.501625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.501579 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq"] Oct 30 13:55:55.502136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.501857 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" podUID="debfc636-fedd-4960-bc1a-77b6f74e8e88" containerName="maas-api" containerID="cri-o://a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211" gracePeriod=30 Oct 30 13:55:55.663835 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.663801 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:55.790366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.790334 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:55:55.862550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.862520 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-metrics-certs\") pod \"debfc636-fedd-4960-bc1a-77b6f74e8e88\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " Oct 30 13:55:55.862718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.862610 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-maas-api-tls\") pod \"debfc636-fedd-4960-bc1a-77b6f74e8e88\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " Oct 30 13:55:55.862718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.862652 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debfc636-fedd-4960-bc1a-77b6f74e8e88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debfc636-fedd-4960-bc1a-77b6f74e8e88-openshift-service-ca\") pod \"debfc636-fedd-4960-bc1a-77b6f74e8e88\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " Oct 30 13:55:55.862718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.862699 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debfc636-fedd-4960-bc1a-77b6f74e8e88-kube-api-access-knjjg\" (UniqueName: \"kubernetes.io/projected/debfc636-fedd-4960-bc1a-77b6f74e8e88-kube-api-access-knjjg\") pod \"debfc636-fedd-4960-bc1a-77b6f74e8e88\" (UID: \"debfc636-fedd-4960-bc1a-77b6f74e8e88\") " Oct 30 13:55:55.863087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.863062 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debfc636-fedd-4960-bc1a-77b6f74e8e88-openshift-service-ca" pod "debfc636-fedd-4960-bc1a-77b6f74e8e88" (UID: "debfc636-fedd-4960-bc1a-77b6f74e8e88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:55:55.863200 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.863181 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debfc636-fedd-4960-bc1a-77b6f74e8e88-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:55.865046 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.865022 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-metrics-certs" pod "debfc636-fedd-4960-bc1a-77b6f74e8e88" (UID: "debfc636-fedd-4960-bc1a-77b6f74e8e88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:55.865163 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.865098 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debfc636-fedd-4960-bc1a-77b6f74e8e88-kube-api-access-knjjg" pod "debfc636-fedd-4960-bc1a-77b6f74e8e88" (UID: "debfc636-fedd-4960-bc1a-77b6f74e8e88"). InnerVolumeSpecName "kube-api-access-knjjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:55:55.865359 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.865340 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-maas-api-tls" pod "debfc636-fedd-4960-bc1a-77b6f74e8e88" (UID: "debfc636-fedd-4960-bc1a-77b6f74e8e88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:55:55.988886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.988841 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:55.988886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.988876 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knjjg\" (UniqueName: \"kubernetes.io/projected/debfc636-fedd-4960-bc1a-77b6f74e8e88-kube-api-access-knjjg\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:55.988886 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:55.988890 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debfc636-fedd-4960-bc1a-77b6f74e8e88-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:55:56.005029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.004992 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" path="/var/lib/kubelet/pods/04d0d438-e1e1-434a-9b86-abf3f11e7f2b/volumes" Oct 30 13:55:56.191376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.191344 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:56.395765 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.395725 5515 generic.go:356] "Generic (PLEG): container finished" podID="debfc636-fedd-4960-bc1a-77b6f74e8e88" containerID="a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211" exitCode=0 Oct 30 13:55:56.395933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.395772 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" Oct 30 13:55:56.395933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.395816 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" event={"ID":"debfc636-fedd-4960-bc1a-77b6f74e8e88","Type":"ContainerDied","Data":"a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211"} Oct 30 13:55:56.395933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.395874 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq" event={"ID":"debfc636-fedd-4960-bc1a-77b6f74e8e88","Type":"ContainerDied","Data":"d9a50dd7c518d28ff7a51be9eb5d058166e6fd9f4475091e24b8153bd45a1e59"} Oct 30 13:55:56.395933 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.395901 5515 scope.go:122] "RemoveContainer" containerID="a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211" Oct 30 13:55:56.402812 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.402786 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:56.405845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.405826 5515 scope.go:122] "RemoveContainer" containerID="a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211" Oct 30 13:55:56.406115 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:56.406095 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211\": container with ID starting with a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211 not found: ID does not exist" containerID="a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211" Oct 30 13:55:56.406193 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.406128 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211"} err="failed to get container status \"a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211\": rpc error: code = NotFound desc = could not find container \"a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211\": container with ID starting with a47572ef41a88b31b47ec4b407a4a8a0d4f9be471a7321b93cb7f65ccd343211 not found: ID does not exist" Oct 30 13:55:56.447232 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.447200 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq"] Oct 30 13:55:56.459426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.459399 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4db150-fdcf6dd6c-nrnnq"] Oct 30 13:55:56.838152 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:56.838057 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:58.010310 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:58.010259 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debfc636-fedd-4960-bc1a-77b6f74e8e88" path="/var/lib/kubelet/pods/debfc636-fedd-4960-bc1a-77b6f74e8e88/volumes" Oct 30 13:55:58.103374 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:58.103341 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:58.988753 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:55:58.988718 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:55:59.997315 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:55:59.997062 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:00.206062 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:00.206023 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:00.490215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:00.490179 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:00.727492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:00.727460 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:00.931057 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:00.931024 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:01.149704 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:01.149673 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:01.801195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:01.801136 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:03.295594 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:03.295562 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:03.974620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:03.974584 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:04.177094 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:04.177058 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz"] Oct 30 13:56:04.177384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:04.177348 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" containerID="cri-o://ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266" gracePeriod=30 Oct 30 13:56:04.187807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:04.187777 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:04.395509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:04.395475 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:04.967563 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:04.967524 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:06.397988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.397955 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp"] Oct 30 13:56:06.398507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398493 5515 cpu_manager.go:414] "containerMap: removing container" podUID="debfc636-fedd-4960-bc1a-77b6f74e8e88" containerName="maas-api" Oct 30 13:56:06.398559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398509 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debfc636-fedd-4960-bc1a-77b6f74e8e88" containerName="maas-api" Oct 30 13:56:06.398559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398523 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" containerName="payload-pre-processing" Oct 30 13:56:06.398559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398529 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" containerName="payload-pre-processing" Oct 30 13:56:06.398559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398537 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" Oct 30 13:56:06.398559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398542 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" Oct 30 13:56:06.398559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398549 5515 cpu_manager.go:414] "containerMap: removing container" podUID="6b564df0-7119-482f-8404-9a78078481aa" containerName="maas-api" Oct 30 13:56:06.398559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398553 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b564df0-7119-482f-8404-9a78078481aa" containerName="maas-api" Oct 30 13:56:06.398559 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398561 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ea2cbef4-f022-4196-afac-643eda0f7855" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398565 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2cbef4-f022-4196-afac-643eda0f7855" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398578 5515 cpu_manager.go:414] "containerMap: removing container" podUID="07a4146d-bbb6-48c8-b47e-4372fdc387cc" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398583 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a4146d-bbb6-48c8-b47e-4372fdc387cc" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398591 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f1c02cc8-b166-4d7a-911d-1f731e97f461" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398596 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c02cc8-b166-4d7a-911d-1f731e97f461" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398603 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ff901dc4-d932-4a95-8e36-e1ccc2ab4050" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398609 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff901dc4-d932-4a95-8e36-e1ccc2ab4050" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398620 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9a1973c8-eb1b-471e-a568-b81943e66597" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398625 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1973c8-eb1b-471e-a568-b81943e66597" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398630 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398634 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398640 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398644 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398654 5515 cpu_manager.go:414] "containerMap: removing container" podUID="60e64645-cc47-496b-980b-f22d6ed896e7" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398659 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e64645-cc47-496b-980b-f22d6ed896e7" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398667 5515 cpu_manager.go:414] "containerMap: removing container" podUID="85074b34-4622-4d92-a481-2075120a066c" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398671 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85074b34-4622-4d92-a481-2075120a066c" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398680 5515 cpu_manager.go:414] "containerMap: removing container" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398684 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398691 5515 cpu_manager.go:414] "containerMap: removing container" podUID="488a7efe-889c-4c6d-a9a1-5fee8a394212" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398695 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488a7efe-889c-4c6d-a9a1-5fee8a394212" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398706 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ad4c9f3d-fa30-4061-9ce5-9e557722441f" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398710 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4c9f3d-fa30-4061-9ce5-9e557722441f" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398719 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398725 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398732 5515 cpu_manager.go:414] "containerMap: removing container" podUID="5856c551-dc30-4e8f-a684-74c4e564b3a8" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398736 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5856c551-dc30-4e8f-a684-74c4e564b3a8" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398742 5515 cpu_manager.go:414] "containerMap: removing container" podUID="fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398747 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398754 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398758 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398766 5515 cpu_manager.go:414] "containerMap: removing container" podUID="080d1adc-43f1-4344-97a4-618a8b5ea618" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398770 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080d1adc-43f1-4344-97a4-618a8b5ea618" containerName="maas-api" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398776 5515 cpu_manager.go:414] "containerMap: removing container" podUID="a3b8ed37-561a-4a4f-8ad1-74a065862db4" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398780 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b8ed37-561a-4a4f-8ad1-74a065862db4" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398785 5515 cpu_manager.go:414] "containerMap: removing container" podUID="63cee96e-e0c8-4f3d-8d2d-6df4243d1469" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398790 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cee96e-e0c8-4f3d-8d2d-6df4243d1469" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398802 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ca2ea174-b297-44ab-96ea-ee1eaf59dbde" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398807 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2ea174-b297-44ab-96ea-ee1eaf59dbde" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398815 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d8b1762a-e274-4dad-9f2c-a2c5f6e58380" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398820 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b1762a-e274-4dad-9f2c-a2c5f6e58380" containerName="payload-pre-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398829 5515 cpu_manager.go:414] "containerMap: removing container" podUID="fa629737-3309-43f6-8d6d-bea2f57918de" containerName="payload-processing" Oct 30 13:56:06.398814 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398833 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa629737-3309-43f6-8d6d-bea2f57918de" containerName="payload-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398843 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398847 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398917 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e64645-cc47-496b-980b-f22d6ed896e7" containerName="payload-pre-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398925 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="85074b34-4622-4d92-a481-2075120a066c" containerName="payload-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398935 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d0d438-e1e1-434a-9b86-abf3f11e7f2b" containerName="istio-proxy" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398943 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="488a7efe-889c-4c6d-a9a1-5fee8a394212" containerName="payload-pre-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398950 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4c9f3d-fa30-4061-9ce5-9e557722441f" containerName="payload-pre-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398956 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398961 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5856c551-dc30-4e8f-a684-74c4e564b3a8" containerName="maas-api" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398966 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc85d4de-d18a-4213-bdb8-4d2f1c198fdd" containerName="payload-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398971 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398977 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="080d1adc-43f1-4344-97a4-618a8b5ea618" containerName="maas-api" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398983 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b8ed37-561a-4a4f-8ad1-74a065862db4" containerName="payload-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398989 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cee96e-e0c8-4f3d-8d2d-6df4243d1469" containerName="payload-pre-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.398995 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2ea174-b297-44ab-96ea-ee1eaf59dbde" containerName="payload-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399000 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b1762a-e274-4dad-9f2c-a2c5f6e58380" containerName="payload-pre-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399005 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa629737-3309-43f6-8d6d-bea2f57918de" containerName="payload-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399011 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="debfc636-fedd-4960-bc1a-77b6f74e8e88" containerName="maas-api" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399017 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dfe827c-ab2e-4d55-9b10-ad2e83dd742e" containerName="payload-pre-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399022 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed15e5d5-20e0-4eb1-8fd0-4905858cbaea" containerName="istio-proxy" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399027 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b564df0-7119-482f-8404-9a78078481aa" containerName="maas-api" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399034 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2cbef4-f022-4196-afac-643eda0f7855" containerName="payload-pre-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399040 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a4146d-bbb6-48c8-b47e-4372fdc387cc" containerName="maas-api" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399046 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c02cc8-b166-4d7a-911d-1f731e97f461" containerName="payload-pre-processing" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399051 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff901dc4-d932-4a95-8e36-e1ccc2ab4050" containerName="maas-api" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399057 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1973c8-eb1b-471e-a568-b81943e66597" containerName="maas-api" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399061 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f09afe-2478-4f1c-b9c0-dca74dc14765" containerName="istio-proxy" Oct 30 13:56:06.400146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.399067 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69244e3-72b4-4d7d-b277-1ddd0c0e9de6" containerName="payload-processing" Oct 30 13:56:06.404385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.404365 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.428478 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.428433 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp"] Oct 30 13:56:06.436177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.436065 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:56:06.436626 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.436602 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\"" Oct 30 13:56:06.511699 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.511660 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kd99\" (UniqueName: \"kubernetes.io/projected/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-kube-api-access-5kd99\") pod \"maas-api-revoke-sub-910539a4-frwtp\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.511901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.511711 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-910539a4-frwtp\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.511901 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.511877 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-910539a4-frwtp\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.579287 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.579235 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:06.613289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.613235 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-910539a4-frwtp\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.613488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.613441 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kd99\" (UniqueName: \"kubernetes.io/projected/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-kube-api-access-5kd99\") pod \"maas-api-revoke-sub-910539a4-frwtp\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.613488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.613484 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-910539a4-frwtp\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.614022 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.613977 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-910539a4-frwtp\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.616192 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.616166 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-910539a4-frwtp\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.637256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.637226 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kd99\" (UniqueName: \"kubernetes.io/projected/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-kube-api-access-5kd99\") pod \"maas-api-revoke-sub-910539a4-frwtp\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:06.722391 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:06.722313 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:07.022840 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:07.022814 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp"] Oct 30 13:56:07.023379 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:56:07.023354 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89a6cc5_1098_4fbf_baa4_0d2f47200d28.slice/crio-67d0455d5f5d725e22082832c2e04e6353faf4ac5d3acd648ddc1f30393555b0 WatchSource:0}: Error finding container 67d0455d5f5d725e22082832c2e04e6353faf4ac5d3acd648ddc1f30393555b0: Status 404 returned error can't find the container with id 67d0455d5f5d725e22082832c2e04e6353faf4ac5d3acd648ddc1f30393555b0 Oct 30 13:56:07.197398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:07.197371 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:07.465586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:07.465550 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" event={"ID":"c89a6cc5-1098-4fbf-baa4-0d2f47200d28","Type":"ContainerStarted","Data":"6a52568617d3068225da14079a8935cc12a7e1b313b46b6a6c61f96f33b1056b"} Oct 30 13:56:07.466004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:07.465607 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" event={"ID":"c89a6cc5-1098-4fbf-baa4-0d2f47200d28","Type":"ContainerStarted","Data":"67d0455d5f5d725e22082832c2e04e6353faf4ac5d3acd648ddc1f30393555b0"} Oct 30 13:56:07.500762 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:07.500731 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:07.506622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:07.506577 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" podStartSLOduration=1.506564303 podStartE2EDuration="1.506564303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:07.505221355 +0000 UTC m=+4985.632022254" watchObservedRunningTime="2026-10-30 13:56:07.506564303 +0000 UTC m=+4985.633365195" Oct 30 13:56:07.829857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:07.829828 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:08.474444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:08.474404 5515 generic.go:356] "Generic (PLEG): container finished" podID="c89a6cc5-1098-4fbf-baa4-0d2f47200d28" containerID="6a52568617d3068225da14079a8935cc12a7e1b313b46b6a6c61f96f33b1056b" exitCode=0 Oct 30 13:56:08.474863 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:08.474495 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" event={"ID":"c89a6cc5-1098-4fbf-baa4-0d2f47200d28","Type":"ContainerDied","Data":"6a52568617d3068225da14079a8935cc12a7e1b313b46b6a6c61f96f33b1056b"} Oct 30 13:56:08.755457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:08.755368 5515 patch_prober.go:29] interesting pod/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:56:08.755457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:08.755426 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:56:09.008384 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.008303 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:09.444546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.444517 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:56:09.482001 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.481963 5515 generic.go:356] "Generic (PLEG): container finished" podID="0890391b-13c2-422d-90cf-3ac31659e601" containerID="ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266" exitCode=0 Oct 30 13:56:09.482001 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.481998 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" event={"ID":"0890391b-13c2-422d-90cf-3ac31659e601","Type":"ContainerDied","Data":"ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266"} Oct 30 13:56:09.482507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.482018 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" Oct 30 13:56:09.482507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.482059 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz" event={"ID":"0890391b-13c2-422d-90cf-3ac31659e601","Type":"ContainerDied","Data":"468313e4863f12f7a50f501bf7b5febb1c177ab67f3b9b7d62fa704a3c640d5c"} Oct 30 13:56:09.482507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.482088 5515 scope.go:122] "RemoveContainer" containerID="ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266" Oct 30 13:56:09.491614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.491594 5515 scope.go:122] "RemoveContainer" containerID="eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95" Oct 30 13:56:09.500633 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.500609 5515 scope.go:122] "RemoveContainer" containerID="ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266" Oct 30 13:56:09.500959 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:09.500936 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266\": container with ID starting with ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266 not found: ID does not exist" containerID="ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266" Oct 30 13:56:09.501038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.500967 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266"} err="failed to get container status \"ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266\": rpc error: code = NotFound desc = could not find container \"ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266\": container with ID starting with ad852b10a27dcbaf3d05e9a29b7f8b54dba734087933a1a6f5e0b2df993ba266 not found: ID does not exist" Oct 30 13:56:09.501038 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.500986 5515 scope.go:122] "RemoveContainer" containerID="eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95" Oct 30 13:56:09.501243 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:09.501223 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95\": container with ID starting with eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95 not found: ID does not exist" containerID="eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95" Oct 30 13:56:09.501331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.501249 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95"} err="failed to get container status \"eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95\": rpc error: code = NotFound desc = could not find container \"eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95\": container with ID starting with eda9b0175f14434ea0ad4a1b3b8f0d6de0247d8032576b013fa8368b8fa9ae95 not found: ID does not exist" Oct 30 13:56:09.546828 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.546761 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-credential-socket\") pod \"0890391b-13c2-422d-90cf-3ac31659e601\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " Oct 30 13:56:09.546828 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.546812 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-socket\") pod \"0890391b-13c2-422d-90cf-3ac31659e601\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " Oct 30 13:56:09.547035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.546842 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-envoy\") pod \"0890391b-13c2-422d-90cf-3ac31659e601\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " Oct 30 13:56:09.547035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.546865 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-data\") pod \"0890391b-13c2-422d-90cf-3ac31659e601\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " Oct 30 13:56:09.547035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.546913 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0890391b-13c2-422d-90cf-3ac31659e601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0890391b-13c2-422d-90cf-3ac31659e601-istiod-ca-cert\") pod \"0890391b-13c2-422d-90cf-3ac31659e601\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " Oct 30 13:56:09.547035 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.547007 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-kube-api-access-s9t9h\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-kube-api-access-s9t9h\") pod \"0890391b-13c2-422d-90cf-3ac31659e601\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " Oct 30 13:56:09.547904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.547081 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-credential-socket" pod "0890391b-13c2-422d-90cf-3ac31659e601" (UID: "0890391b-13c2-422d-90cf-3ac31659e601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:09.547904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.547089 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-certs\") pod \"0890391b-13c2-422d-90cf-3ac31659e601\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " Oct 30 13:56:09.547904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.547196 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0890391b-13c2-422d-90cf-3ac31659e601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0890391b-13c2-422d-90cf-3ac31659e601-istio-podinfo\") pod \"0890391b-13c2-422d-90cf-3ac31659e601\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " Oct 30 13:56:09.547904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.547311 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-socket" pod "0890391b-13c2-422d-90cf-3ac31659e601" (UID: "0890391b-13c2-422d-90cf-3ac31659e601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:09.547904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.547358 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-data" pod "0890391b-13c2-422d-90cf-3ac31659e601" (UID: "0890391b-13c2-422d-90cf-3ac31659e601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:09.547904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.547508 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0890391b-13c2-422d-90cf-3ac31659e601-istiod-ca-cert" pod "0890391b-13c2-422d-90cf-3ac31659e601" (UID: "0890391b-13c2-422d-90cf-3ac31659e601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:09.547904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.547574 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-istio-token\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-istio-token\") pod \"0890391b-13c2-422d-90cf-3ac31659e601\" (UID: \"0890391b-13c2-422d-90cf-3ac31659e601\") " Oct 30 13:56:09.548313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.547940 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-certs" pod "0890391b-13c2-422d-90cf-3ac31659e601" (UID: "0890391b-13c2-422d-90cf-3ac31659e601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:09.548313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.548090 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.548313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.548112 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.548313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.548127 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.548313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.548141 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0890391b-13c2-422d-90cf-3ac31659e601-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.548313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.548153 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.550602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.550580 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-kube-api-access-s9t9h" pod "0890391b-13c2-422d-90cf-3ac31659e601" (UID: "0890391b-13c2-422d-90cf-3ac31659e601"). InnerVolumeSpecName "kube-api-access-s9t9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:09.552011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.551993 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0890391b-13c2-422d-90cf-3ac31659e601-istio-podinfo" pod "0890391b-13c2-422d-90cf-3ac31659e601" (UID: "0890391b-13c2-422d-90cf-3ac31659e601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:56:09.552110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.552008 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-envoy" pod "0890391b-13c2-422d-90cf-3ac31659e601" (UID: "0890391b-13c2-422d-90cf-3ac31659e601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:09.552110 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.552045 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-istio-token" pod "0890391b-13c2-422d-90cf-3ac31659e601" (UID: "0890391b-13c2-422d-90cf-3ac31659e601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:09.629558 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.629532 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:09.649130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.649096 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0890391b-13c2-422d-90cf-3ac31659e601-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.649130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.649127 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9t9h\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-kube-api-access-s9t9h\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.649351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.649141 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0890391b-13c2-422d-90cf-3ac31659e601-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.649351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.649154 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0890391b-13c2-422d-90cf-3ac31659e601-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.749807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.749774 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-xks-ca\") pod \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " Oct 30 13:56:09.750013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.749832 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-openshift-ca\") pod \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " Oct 30 13:56:09.750013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.749876 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-kube-api-access-5kd99\" (UniqueName: \"kubernetes.io/projected/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-kube-api-access-5kd99\") pod \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\" (UID: \"c89a6cc5-1098-4fbf-baa4-0d2f47200d28\") " Oct 30 13:56:09.750412 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.750234 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-openshift-ca" pod "c89a6cc5-1098-4fbf-baa4-0d2f47200d28" (UID: "c89a6cc5-1098-4fbf-baa4-0d2f47200d28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:09.752033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.752009 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-kube-api-access-5kd99" pod "c89a6cc5-1098-4fbf-baa4-0d2f47200d28" (UID: "c89a6cc5-1098-4fbf-baa4-0d2f47200d28"). InnerVolumeSpecName "kube-api-access-5kd99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:09.752033 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.752025 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-xks-ca" pod "c89a6cc5-1098-4fbf-baa4-0d2f47200d28" (UID: "c89a6cc5-1098-4fbf-baa4-0d2f47200d28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:56:09.828396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.828307 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz"] Oct 30 13:56:09.832177 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.832144 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-4db150-openshift-default-75fddffd6f-c4pkz"] Oct 30 13:56:09.851810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.851778 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kd99\" (UniqueName: \"kubernetes.io/projected/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-kube-api-access-5kd99\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.851810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.851804 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-xks-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:09.851810 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:09.851816 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89a6cc5-1098-4fbf-baa4-0d2f47200d28-maas-api-openshift-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:10.002161 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:10.002118 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0890391b-13c2-422d-90cf-3ac31659e601" path="/var/lib/kubelet/pods/0890391b-13c2-422d-90cf-3ac31659e601/volumes" Oct 30 13:56:10.489532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:10.489438 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" Oct 30 13:56:10.489532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:10.489450 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910539a4-frwtp" event={"ID":"c89a6cc5-1098-4fbf-baa4-0d2f47200d28","Type":"ContainerDied","Data":"67d0455d5f5d725e22082832c2e04e6353faf4ac5d3acd648ddc1f30393555b0"} Oct 30 13:56:10.489532 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:10.489500 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d0455d5f5d725e22082832c2e04e6353faf4ac5d3acd648ddc1f30393555b0" Oct 30 13:56:10.561842 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:10.561814 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:11.217245 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:11.217206 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:11.491332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:11.491222 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:11.832010 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:11.831921 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:13.017343 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:13.017308 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:14.747117 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:14.747082 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:14.962923 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:14.962855 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:14.973080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:14.973047 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd"] Oct 30 13:56:14.973351 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:14.973324 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" containerID="cri-o://13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d" gracePeriod=30 Oct 30 13:56:15.171751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:15.171717 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:15.242854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:15.242819 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8"] Oct 30 13:56:15.491981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:15.491891 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:15.698501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:15.698460 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:16.500724 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:16.500687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:17.240755 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:17.240721 5515 patch_prober.go:29] interesting pod/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:56:17.240932 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:17.240792 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:56:17.425649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:17.425620 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:18.390692 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:18.390657 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:18.821622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:18.821531 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:19.010962 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:19.010932 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn"] Oct 30 13:56:19.011160 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:19.011121 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" containerID="cri-o://a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976" gracePeriod=30 Oct 30 13:56:19.035638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:19.035611 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:19.247904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:19.247874 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:20.189342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.189309 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:20.241924 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.241854 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:56:20.243528 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.243497 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" podUID="60b7cce7-5160-459f-b487-a0c3074697cd" containerName="payload-processing" containerID="cri-o://e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1" gracePeriod=25 Oct 30 13:56:20.379129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379103 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-socket\") pod \"e8a66893-8ec4-4286-b2de-52c5202295de\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " Oct 30 13:56:20.379333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379141 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-kube-api-access-f6ts6\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-kube-api-access-f6ts6\") pod \"e8a66893-8ec4-4286-b2de-52c5202295de\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " Oct 30 13:56:20.379333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379208 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-data\") pod \"e8a66893-8ec4-4286-b2de-52c5202295de\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " Oct 30 13:56:20.379333 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379316 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a66893-8ec4-4286-b2de-52c5202295de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a66893-8ec4-4286-b2de-52c5202295de-istiod-ca-cert\") pod \"e8a66893-8ec4-4286-b2de-52c5202295de\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " Oct 30 13:56:20.379510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379365 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8a66893-8ec4-4286-b2de-52c5202295de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a66893-8ec4-4286-b2de-52c5202295de-istio-podinfo\") pod \"e8a66893-8ec4-4286-b2de-52c5202295de\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " Oct 30 13:56:20.379510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379399 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-socket" pod "e8a66893-8ec4-4286-b2de-52c5202295de" (UID: "e8a66893-8ec4-4286-b2de-52c5202295de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:20.379510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379407 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-credential-socket\") pod \"e8a66893-8ec4-4286-b2de-52c5202295de\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " Oct 30 13:56:20.379687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379502 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-data" pod "e8a66893-8ec4-4286-b2de-52c5202295de" (UID: "e8a66893-8ec4-4286-b2de-52c5202295de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:20.379687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379505 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-envoy\") pod \"e8a66893-8ec4-4286-b2de-52c5202295de\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " Oct 30 13:56:20.379687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379634 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-certs\") pod \"e8a66893-8ec4-4286-b2de-52c5202295de\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " Oct 30 13:56:20.379687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379670 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-istio-token\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-istio-token\") pod \"e8a66893-8ec4-4286-b2de-52c5202295de\" (UID: \"e8a66893-8ec4-4286-b2de-52c5202295de\") " Oct 30 13:56:20.379915 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379667 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-credential-socket" pod "e8a66893-8ec4-4286-b2de-52c5202295de" (UID: "e8a66893-8ec4-4286-b2de-52c5202295de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:20.379915 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379775 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a66893-8ec4-4286-b2de-52c5202295de-istiod-ca-cert" pod "e8a66893-8ec4-4286-b2de-52c5202295de" (UID: "e8a66893-8ec4-4286-b2de-52c5202295de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:20.379915 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.379875 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-certs" pod "e8a66893-8ec4-4286-b2de-52c5202295de" (UID: "e8a66893-8ec4-4286-b2de-52c5202295de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:20.380219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.380188 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:20.380219 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.380216 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:20.380381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.380230 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:20.380381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.380242 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:20.380381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.380255 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a66893-8ec4-4286-b2de-52c5202295de-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:20.381634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.381602 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-istio-token" pod "e8a66893-8ec4-4286-b2de-52c5202295de" (UID: "e8a66893-8ec4-4286-b2de-52c5202295de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:20.381634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.381628 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-envoy" pod "e8a66893-8ec4-4286-b2de-52c5202295de" (UID: "e8a66893-8ec4-4286-b2de-52c5202295de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:20.381948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.381932 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-kube-api-access-f6ts6" pod "e8a66893-8ec4-4286-b2de-52c5202295de" (UID: "e8a66893-8ec4-4286-b2de-52c5202295de"). InnerVolumeSpecName "kube-api-access-f6ts6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:20.381948 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.381936 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8a66893-8ec4-4286-b2de-52c5202295de-istio-podinfo" pod "e8a66893-8ec4-4286-b2de-52c5202295de" (UID: "e8a66893-8ec4-4286-b2de-52c5202295de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:56:20.447640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.447605 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv"] Oct 30 13:56:20.457617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.457589 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb"] Oct 30 13:56:20.481257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.481230 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a66893-8ec4-4286-b2de-52c5202295de-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:20.481257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.481255 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:20.481453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.481277 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6ts6\" (UniqueName: \"kubernetes.io/projected/e8a66893-8ec4-4286-b2de-52c5202295de-kube-api-access-f6ts6\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:20.481453 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.481289 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a66893-8ec4-4286-b2de-52c5202295de-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:20.549289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.549195 5515 generic.go:356] "Generic (PLEG): container finished" podID="e8a66893-8ec4-4286-b2de-52c5202295de" containerID="13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d" exitCode=0 Oct 30 13:56:20.549289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.549253 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" Oct 30 13:56:20.549546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.549294 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" event={"ID":"e8a66893-8ec4-4286-b2de-52c5202295de","Type":"ContainerDied","Data":"13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d"} Oct 30 13:56:20.549546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.549363 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd" event={"ID":"e8a66893-8ec4-4286-b2de-52c5202295de","Type":"ContainerDied","Data":"fcf9c0a8e0e98610e68dd757352eff08d049a285e9cb391657e9bfbd405a55ef"} Oct 30 13:56:20.549546 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.549393 5515 scope.go:122] "RemoveContainer" containerID="13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d" Oct 30 13:56:20.559640 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.559621 5515 scope.go:122] "RemoveContainer" containerID="47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc" Oct 30 13:56:20.568490 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.568469 5515 scope.go:122] "RemoveContainer" containerID="13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d" Oct 30 13:56:20.568772 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:20.568753 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d\": container with ID starting with 13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d not found: ID does not exist" containerID="13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d" Oct 30 13:56:20.568818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.568781 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d"} err="failed to get container status \"13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d\": rpc error: code = NotFound desc = could not find container \"13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d\": container with ID starting with 13069d57f361465bbb769cc00feb18dd40a5cb0149085b73aa5ce558cce2713d not found: ID does not exist" Oct 30 13:56:20.568818 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.568798 5515 scope.go:122] "RemoveContainer" containerID="47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc" Oct 30 13:56:20.569051 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:20.569018 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc\": container with ID starting with 47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc not found: ID does not exist" containerID="47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc" Oct 30 13:56:20.569119 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.569061 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc"} err="failed to get container status \"47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc\": rpc error: code = NotFound desc = could not find container \"47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc\": container with ID starting with 47d0ff6540eeb5cde6609772f86a0798a77b135c741d9a6775cde867a9d248dc not found: ID does not exist" Oct 30 13:56:20.603815 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.603788 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd"] Oct 30 13:56:20.607417 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.607392 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92855c-openshift-default-7f5cd54b5c-kcrcd"] Oct 30 13:56:20.654586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:20.654559 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:21.628357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:21.628323 5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:56:21.628776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:21.628376 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:56:21.729175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:21.729140 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:22.001480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:22.001440 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" path="/var/lib/kubelet/pods/e8a66893-8ec4-4286-b2de-52c5202295de/volumes" Oct 30 13:56:22.288802 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:22.288715 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:22.505878 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:22.505839 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:22.945781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:22.945750 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:23.840240 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:23.840201 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:24.057116 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.057080 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:24.281411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.281374 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:56:24.299616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.299583 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:24.429027 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.428999 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-socket\") pod \"deb79059-60be-46d7-a8d5-249714d32aab\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " Oct 30 13:56:24.429237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429048 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-istio-token\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-istio-token\") pod \"deb79059-60be-46d7-a8d5-249714d32aab\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " Oct 30 13:56:24.429237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429092 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-kube-api-access-h72mx\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-kube-api-access-h72mx\") pod \"deb79059-60be-46d7-a8d5-249714d32aab\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " Oct 30 13:56:24.429237 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429134 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-envoy\") pod \"deb79059-60be-46d7-a8d5-249714d32aab\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " Oct 30 13:56:24.429494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429255 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-certs\") pod \"deb79059-60be-46d7-a8d5-249714d32aab\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " Oct 30 13:56:24.429494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429306 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-data\") pod \"deb79059-60be-46d7-a8d5-249714d32aab\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " Oct 30 13:56:24.429494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429346 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-socket" pod "deb79059-60be-46d7-a8d5-249714d32aab" (UID: "deb79059-60be-46d7-a8d5-249714d32aab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:24.429494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429377 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb79059-60be-46d7-a8d5-249714d32aab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb79059-60be-46d7-a8d5-249714d32aab-istiod-ca-cert\") pod \"deb79059-60be-46d7-a8d5-249714d32aab\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " Oct 30 13:56:24.429494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429473 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-credential-socket\") pod \"deb79059-60be-46d7-a8d5-249714d32aab\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " Oct 30 13:56:24.429809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429517 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deb79059-60be-46d7-a8d5-249714d32aab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb79059-60be-46d7-a8d5-249714d32aab-istio-podinfo\") pod \"deb79059-60be-46d7-a8d5-249714d32aab\" (UID: \"deb79059-60be-46d7-a8d5-249714d32aab\") " Oct 30 13:56:24.429809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429590 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-certs" pod "deb79059-60be-46d7-a8d5-249714d32aab" (UID: "deb79059-60be-46d7-a8d5-249714d32aab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:24.429922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429807 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-credential-socket" pod "deb79059-60be-46d7-a8d5-249714d32aab" (UID: "deb79059-60be-46d7-a8d5-249714d32aab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:24.429976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.429921 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb79059-60be-46d7-a8d5-249714d32aab-istiod-ca-cert" pod "deb79059-60be-46d7-a8d5-249714d32aab" (UID: "deb79059-60be-46d7-a8d5-249714d32aab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:24.430061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.430043 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:24.430134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.430066 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:24.430134 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.430081 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:24.430239 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.430141 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-data" pod "deb79059-60be-46d7-a8d5-249714d32aab" (UID: "deb79059-60be-46d7-a8d5-249714d32aab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:24.431601 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.431580 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-istio-token" pod "deb79059-60be-46d7-a8d5-249714d32aab" (UID: "deb79059-60be-46d7-a8d5-249714d32aab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:24.431781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.431758 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-envoy" pod "deb79059-60be-46d7-a8d5-249714d32aab" (UID: "deb79059-60be-46d7-a8d5-249714d32aab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:24.432024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.432001 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deb79059-60be-46d7-a8d5-249714d32aab-istio-podinfo" pod "deb79059-60be-46d7-a8d5-249714d32aab" (UID: "deb79059-60be-46d7-a8d5-249714d32aab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:56:24.432100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.432019 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-kube-api-access-h72mx" pod "deb79059-60be-46d7-a8d5-249714d32aab" (UID: "deb79059-60be-46d7-a8d5-249714d32aab"). InnerVolumeSpecName "kube-api-access-h72mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:24.519100 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.519066 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" podUID="60b7cce7-5160-459f-b487-a0c3074697cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:56:24.531316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.531288 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb79059-60be-46d7-a8d5-249714d32aab-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:24.531316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.531311 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:24.531316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.531321 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h72mx\" (UniqueName: \"kubernetes.io/projected/deb79059-60be-46d7-a8d5-249714d32aab-kube-api-access-h72mx\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:24.531512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.531329 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:24.531512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.531337 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb79059-60be-46d7-a8d5-249714d32aab-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:24.531512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.531346 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb79059-60be-46d7-a8d5-249714d32aab-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:24.573460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.573428 5515 generic.go:356] "Generic (PLEG): container finished" podID="deb79059-60be-46d7-a8d5-249714d32aab" containerID="a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976" exitCode=0 Oct 30 13:56:24.573628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.573478 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" Oct 30 13:56:24.573628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.573504 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" event={"ID":"deb79059-60be-46d7-a8d5-249714d32aab","Type":"ContainerDied","Data":"a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976"} Oct 30 13:56:24.573628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.573557 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn" event={"ID":"deb79059-60be-46d7-a8d5-249714d32aab","Type":"ContainerDied","Data":"c7ba1d6ace25378a0d50c7c678ec48285f9771dc70f68052647c2e1ebf65bdb3"} Oct 30 13:56:24.573628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.573613 5515 scope.go:122] "RemoveContainer" containerID="a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976" Oct 30 13:56:24.584364 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.584339 5515 scope.go:122] "RemoveContainer" containerID="b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334" Oct 30 13:56:24.597420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.597391 5515 scope.go:122] "RemoveContainer" containerID="a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976" Oct 30 13:56:24.597829 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:24.597798 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976\": container with ID starting with a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976 not found: ID does not exist" containerID="a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976" Oct 30 13:56:24.597937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.597842 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976"} err="failed to get container status \"a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976\": rpc error: code = NotFound desc = could not find container \"a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976\": container with ID starting with a0bb38dc7e389b7ef11872f72caa1b9343fef9b44eeaad90346809a4086ce976 not found: ID does not exist" Oct 30 13:56:24.597937 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.597873 5515 scope.go:122] "RemoveContainer" containerID="b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334" Oct 30 13:56:24.598342 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:24.598313 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334\": container with ID starting with b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334 not found: ID does not exist" containerID="b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334" Oct 30 13:56:24.598342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.598350 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334"} err="failed to get container status \"b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334\": rpc error: code = NotFound desc = could not find container \"b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334\": container with ID starting with b8d1559a86092c27e9580026f7a7ceb5a6cff8dc340fd62bd909689e5f3ec334 not found: ID does not exist" Oct 30 13:56:24.633799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.633767 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn"] Oct 30 13:56:24.635610 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:24.635589 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-517a25-openshift-default-66996c9485-55wzn"] Oct 30 13:56:25.490634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:25.490601 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:56:25.988299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:25.988248 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:26.001504 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:26.001477 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb79059-60be-46d7-a8d5-249714d32aab" path="/var/lib/kubelet/pods/deb79059-60be-46d7-a8d5-249714d32aab/volumes" Oct 30 13:56:26.190889 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:26.190849 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg"] Oct 30 13:56:26.500451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:26.500418 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:26.709991 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:26.709947 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:27.402563 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.402528 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c"] Oct 30 13:56:27.402845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.402795 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" podUID="0bacc714-649c-4d1b-9ee5-fce729d9d7ef" containerName="maas-api" containerID="cri-o://8625d9fbe1d1365798ce95063671f81fec0c5190e78205c10c375b4752cec5a1" gracePeriod=30 Oct 30 13:56:27.593175 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.593146 5515 generic.go:356] "Generic (PLEG): container finished" podID="0bacc714-649c-4d1b-9ee5-fce729d9d7ef" containerID="8625d9fbe1d1365798ce95063671f81fec0c5190e78205c10c375b4752cec5a1" exitCode=0 Oct 30 13:56:27.593544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.593203 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" event={"ID":"0bacc714-649c-4d1b-9ee5-fce729d9d7ef","Type":"ContainerDied","Data":"8625d9fbe1d1365798ce95063671f81fec0c5190e78205c10c375b4752cec5a1"} Oct 30 13:56:27.662452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.662393 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:56:27.671733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.671704 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-maas-api-tls\") pod \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " Oct 30 13:56:27.671862 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.671823 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-openshift-service-ca\") pod \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " Oct 30 13:56:27.671908 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.671868 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-metrics-certs\") pod \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " Oct 30 13:56:27.671943 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.671906 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-kube-api-access-mj96p\" (UniqueName: \"kubernetes.io/projected/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-kube-api-access-mj96p\") pod \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\" (UID: \"0bacc714-649c-4d1b-9ee5-fce729d9d7ef\") " Oct 30 13:56:27.672228 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.672199 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-openshift-service-ca" pod "0bacc714-649c-4d1b-9ee5-fce729d9d7ef" (UID: "0bacc714-649c-4d1b-9ee5-fce729d9d7ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:27.672394 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.672377 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:27.673988 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.673967 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-metrics-certs" pod "0bacc714-649c-4d1b-9ee5-fce729d9d7ef" (UID: "0bacc714-649c-4d1b-9ee5-fce729d9d7ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:56:27.674332 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.674310 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-maas-api-tls" pod "0bacc714-649c-4d1b-9ee5-fce729d9d7ef" (UID: "0bacc714-649c-4d1b-9ee5-fce729d9d7ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:56:27.674420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.674345 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-kube-api-access-mj96p" pod "0bacc714-649c-4d1b-9ee5-fce729d9d7ef" (UID: "0bacc714-649c-4d1b-9ee5-fce729d9d7ef"). InnerVolumeSpecName "kube-api-access-mj96p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:27.773205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.773159 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:27.773205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.773200 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj96p\" (UniqueName: \"kubernetes.io/projected/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-kube-api-access-mj96p\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:27.773205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.773212 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bacc714-649c-4d1b-9ee5-fce729d9d7ef-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:27.869146 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:27.869110 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:28.092577 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:28.092545 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:28.309792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:28.309710 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:28.523360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:28.523328 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:28.608569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:28.608484 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" Oct 30 13:56:28.609064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:28.608455 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c" event={"ID":"0bacc714-649c-4d1b-9ee5-fce729d9d7ef","Type":"ContainerDied","Data":"4616f5dafbd38da8a587ee1b0ef8d0bf47bc583ff6bc61a255e38f938aee3763"} Oct 30 13:56:28.609064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:28.608796 5515 scope.go:122] "RemoveContainer" containerID="8625d9fbe1d1365798ce95063671f81fec0c5190e78205c10c375b4752cec5a1" Oct 30 13:56:28.665738 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:28.665707 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c"] Oct 30 13:56:28.679204 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:28.679168 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-75d385-64cd96bbc8-q445c"] Oct 30 13:56:28.857049 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:28.857017 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:28.987787 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:28.987746 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:56:30.002012 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:30.001979 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bacc714-649c-4d1b-9ee5-fce729d9d7ef" path="/var/lib/kubelet/pods/0bacc714-649c-4d1b-9ee5-fce729d9d7ef/volumes" Oct 30 13:56:30.074867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:30.074823 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4"] Oct 30 13:56:30.102698 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:30.102658 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:30.324667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:30.324585 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:30.542920 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:30.542885 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:31.169727 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:31.169696 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:31.192135 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:31.192094 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" podUID="c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" containerName="payload-processing" containerID="cri-o://035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b" gracePeriod=25 Oct 30 13:56:31.409672 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:31.409638 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:31.457921 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:31.457841 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj"] Oct 30 13:56:31.458083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:31.458036 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" containerID="cri-o://aa74bb58d202ab9521bf686b9c3cd7796248f0bd5761b2861fdb3165beef11d3" gracePeriod=30 Oct 30 13:56:31.626181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:31.626144 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:31.843545 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:31.843444 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:33.833181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:33.833129 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:34.519955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:34.519915 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" podUID="60b7cce7-5160-459f-b487-a0c3074697cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:56:35.075599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.075555 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" podUID="3d828ed9-4e9f-4194-a442-bd8f9059be68" containerName="payload-processing" containerID="cri-o://22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb" gracePeriod=25 Oct 30 13:56:35.357751 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.357656 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:35.468690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.468624 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" containerID="cri-o://9a7a6ed7152867ac1b94d34471c75c51ca32f8bee599023a8d9f897b4c2bee8e" gracePeriod=45 Oct 30 13:56:35.477449 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.477405 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" containerID="cri-o://e1f907b9dec7518c0c3220d868372b5a30a0ee3ddc80a2b2b941e12907506d67" gracePeriod=45 Oct 30 13:56:35.566851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.566810 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:35.639414 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.639378 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" podUID="3d828ed9-4e9f-4194-a442-bd8f9059be68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:56:35.653313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.653280 5515 generic.go:356] "Generic (PLEG): container finished" podID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerID="9a7a6ed7152867ac1b94d34471c75c51ca32f8bee599023a8d9f897b4c2bee8e" exitCode=0 Oct 30 13:56:35.653469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.653358 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" event={"ID":"91f6d43e-ecc0-47cb-890e-c1958dc967f2","Type":"ContainerDied","Data":"9a7a6ed7152867ac1b94d34471c75c51ca32f8bee599023a8d9f897b4c2bee8e"} Oct 30 13:56:35.656637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.656602 5515 generic.go:356] "Generic (PLEG): container finished" podID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerID="e1f907b9dec7518c0c3220d868372b5a30a0ee3ddc80a2b2b941e12907506d67" exitCode=0 Oct 30 13:56:35.656775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.656746 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" event={"ID":"72bf3958-6eff-4b10-9d14-9ae43712b46d","Type":"ContainerDied","Data":"e1f907b9dec7518c0c3220d868372b5a30a0ee3ddc80a2b2b941e12907506d67"} Oct 30 13:56:35.735147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.735117 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 13:56:35.757337 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.757311 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 13:56:35.761181 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761159 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-dshm\") pod \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " Oct 30 13:56:35.761342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761221 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tls-certs\") pod \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " Oct 30 13:56:35.761342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761298 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f6d43e-ecc0-47cb-890e-c1958dc967f2-kube-api-access-dxbzp\" (UniqueName: \"kubernetes.io/projected/91f6d43e-ecc0-47cb-890e-c1958dc967f2-kube-api-access-dxbzp\") pod \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " Oct 30 13:56:35.761342 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761329 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tmp-dir\") pod \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " Oct 30 13:56:35.761506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761432 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-model-cache\") pod \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " Oct 30 13:56:35.761620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761521 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-home\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-home\") pod \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\" (UID: \"91f6d43e-ecc0-47cb-890e-c1958dc967f2\") " Oct 30 13:56:35.761685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761629 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tmp-dir" pod "91f6d43e-ecc0-47cb-890e-c1958dc967f2" (UID: "91f6d43e-ecc0-47cb-890e-c1958dc967f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:35.761685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761649 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-model-cache" pod "91f6d43e-ecc0-47cb-890e-c1958dc967f2" (UID: "91f6d43e-ecc0-47cb-890e-c1958dc967f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:35.761856 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761838 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-home" pod "91f6d43e-ecc0-47cb-890e-c1958dc967f2" (UID: "91f6d43e-ecc0-47cb-890e-c1958dc967f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:35.761970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761956 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.762021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761973 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.762021 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.761983 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.763609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.763587 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f6d43e-ecc0-47cb-890e-c1958dc967f2-kube-api-access-dxbzp" pod "91f6d43e-ecc0-47cb-890e-c1958dc967f2" (UID: "91f6d43e-ecc0-47cb-890e-c1958dc967f2"). InnerVolumeSpecName "kube-api-access-dxbzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:35.763800 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.763782 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-dshm" pod "91f6d43e-ecc0-47cb-890e-c1958dc967f2" (UID: "91f6d43e-ecc0-47cb-890e-c1958dc967f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:35.763867 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.763797 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tls-certs" pod "91f6d43e-ecc0-47cb-890e-c1958dc967f2" (UID: "91f6d43e-ecc0-47cb-890e-c1958dc967f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:56:35.863256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.863219 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bf3958-6eff-4b10-9d14-9ae43712b46d-kube-api-access-z5ddp\" (UniqueName: \"kubernetes.io/projected/72bf3958-6eff-4b10-9d14-9ae43712b46d-kube-api-access-z5ddp\") pod \"72bf3958-6eff-4b10-9d14-9ae43712b46d\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " Oct 30 13:56:35.863470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.863337 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-model-cache\") pod \"72bf3958-6eff-4b10-9d14-9ae43712b46d\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " Oct 30 13:56:35.863470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.863400 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72bf3958-6eff-4b10-9d14-9ae43712b46d-tls-certs\" (UniqueName: \"kubernetes.io/secret/72bf3958-6eff-4b10-9d14-9ae43712b46d-tls-certs\") pod \"72bf3958-6eff-4b10-9d14-9ae43712b46d\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " Oct 30 13:56:35.863777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.863519 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-tmp-dir\") pod \"72bf3958-6eff-4b10-9d14-9ae43712b46d\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " Oct 30 13:56:35.863777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.863561 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-home\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-home\") pod \"72bf3958-6eff-4b10-9d14-9ae43712b46d\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " Oct 30 13:56:35.863777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.863626 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-model-cache" pod "72bf3958-6eff-4b10-9d14-9ae43712b46d" (UID: "72bf3958-6eff-4b10-9d14-9ae43712b46d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:35.863777 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.863675 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-dshm\") pod \"72bf3958-6eff-4b10-9d14-9ae43712b46d\" (UID: \"72bf3958-6eff-4b10-9d14-9ae43712b46d\") " Oct 30 13:56:35.864186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.864118 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f6d43e-ecc0-47cb-890e-c1958dc967f2-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.864186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.864160 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f6d43e-ecc0-47cb-890e-c1958dc967f2-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.864186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.864177 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxbzp\" (UniqueName: \"kubernetes.io/projected/91f6d43e-ecc0-47cb-890e-c1958dc967f2-kube-api-access-dxbzp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.864419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.864191 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.864419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.864117 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-home" pod "72bf3958-6eff-4b10-9d14-9ae43712b46d" (UID: "72bf3958-6eff-4b10-9d14-9ae43712b46d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:35.864419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.864132 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-tmp-dir" pod "72bf3958-6eff-4b10-9d14-9ae43712b46d" (UID: "72bf3958-6eff-4b10-9d14-9ae43712b46d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:35.866060 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.866035 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bf3958-6eff-4b10-9d14-9ae43712b46d-kube-api-access-z5ddp" pod "72bf3958-6eff-4b10-9d14-9ae43712b46d" (UID: "72bf3958-6eff-4b10-9d14-9ae43712b46d"). InnerVolumeSpecName "kube-api-access-z5ddp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:35.866348 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.866327 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bf3958-6eff-4b10-9d14-9ae43712b46d-tls-certs" pod "72bf3958-6eff-4b10-9d14-9ae43712b46d" (UID: "72bf3958-6eff-4b10-9d14-9ae43712b46d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:56:35.866492 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.866470 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-dshm" pod "72bf3958-6eff-4b10-9d14-9ae43712b46d" (UID: "72bf3958-6eff-4b10-9d14-9ae43712b46d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:35.965534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.965444 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.965534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.965479 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.965534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.965490 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bf3958-6eff-4b10-9d14-9ae43712b46d-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.965534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.965502 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5ddp\" (UniqueName: \"kubernetes.io/projected/72bf3958-6eff-4b10-9d14-9ae43712b46d-kube-api-access-z5ddp\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:35.965534 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:35.965514 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bf3958-6eff-4b10-9d14-9ae43712b46d-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.124816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.124784 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:36.341776 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.341687 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:36.558413 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.558381 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:36.666879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.666825 5515 generic.go:356] "Generic (PLEG): container finished" podID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerID="aa74bb58d202ab9521bf686b9c3cd7796248f0bd5761b2861fdb3165beef11d3" exitCode=0 Oct 30 13:56:36.667036 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.666896 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" event={"ID":"ed628cdb-692c-4357-8461-4e8af43a1d5c","Type":"ContainerDied","Data":"aa74bb58d202ab9521bf686b9c3cd7796248f0bd5761b2861fdb3165beef11d3"} Oct 30 13:56:36.669570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.668593 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" event={"ID":"91f6d43e-ecc0-47cb-890e-c1958dc967f2","Type":"ContainerDied","Data":"09ad46c91b6ebf13c1e74ffde58c56c650a072788a33cb8d84667fbad55d9fc8"} Oct 30 13:56:36.669570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.668634 5515 scope.go:122] "RemoveContainer" containerID="9a7a6ed7152867ac1b94d34471c75c51ca32f8bee599023a8d9f897b4c2bee8e" Oct 30 13:56:36.669570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.668756 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv" Oct 30 13:56:36.672444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.672422 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" Oct 30 13:56:36.672444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.672435 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb" event={"ID":"72bf3958-6eff-4b10-9d14-9ae43712b46d","Type":"ContainerDied","Data":"30c6031b2e2a2ca77c939799464f084126b66c66e87bfdfdc008bb7dd6f1c17f"} Oct 30 13:56:36.683098 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.683082 5515 scope.go:122] "RemoveContainer" containerID="e1f907b9dec7518c0c3220d868372b5a30a0ee3ddc80a2b2b941e12907506d67" Oct 30 13:56:36.717833 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.717805 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 13:56:36.774563 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.774530 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-token\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-token\") pod \"ed628cdb-692c-4357-8461-4e8af43a1d5c\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " Oct 30 13:56:36.774721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.774641 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-socket\") pod \"ed628cdb-692c-4357-8461-4e8af43a1d5c\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " Oct 30 13:56:36.774721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.774692 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-envoy\") pod \"ed628cdb-692c-4357-8461-4e8af43a1d5c\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " Oct 30 13:56:36.774830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.774722 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-podinfo\") pod \"ed628cdb-692c-4357-8461-4e8af43a1d5c\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " Oct 30 13:56:36.774830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.774790 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed628cdb-692c-4357-8461-4e8af43a1d5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed628cdb-692c-4357-8461-4e8af43a1d5c-istiod-ca-cert\") pod \"ed628cdb-692c-4357-8461-4e8af43a1d5c\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " Oct 30 13:56:36.774972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.774864 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-data\") pod \"ed628cdb-692c-4357-8461-4e8af43a1d5c\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " Oct 30 13:56:36.774972 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.774943 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-socket" pod "ed628cdb-692c-4357-8461-4e8af43a1d5c" (UID: "ed628cdb-692c-4357-8461-4e8af43a1d5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.775140 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775107 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-data" pod "ed628cdb-692c-4357-8461-4e8af43a1d5c" (UID: "ed628cdb-692c-4357-8461-4e8af43a1d5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.775308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775169 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-kube-api-access-8wgtk\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-kube-api-access-8wgtk\") pod \"ed628cdb-692c-4357-8461-4e8af43a1d5c\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " Oct 30 13:56:36.775308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775205 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-credential-socket\") pod \"ed628cdb-692c-4357-8461-4e8af43a1d5c\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " Oct 30 13:56:36.775308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775211 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed628cdb-692c-4357-8461-4e8af43a1d5c-istiod-ca-cert" pod "ed628cdb-692c-4357-8461-4e8af43a1d5c" (UID: "ed628cdb-692c-4357-8461-4e8af43a1d5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:36.775308 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775237 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-certs\") pod \"ed628cdb-692c-4357-8461-4e8af43a1d5c\" (UID: \"ed628cdb-692c-4357-8461-4e8af43a1d5c\") " Oct 30 13:56:36.775616 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775591 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-credential-socket" pod "ed628cdb-692c-4357-8461-4e8af43a1d5c" (UID: "ed628cdb-692c-4357-8461-4e8af43a1d5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.775691 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775638 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-certs" pod "ed628cdb-692c-4357-8461-4e8af43a1d5c" (UID: "ed628cdb-692c-4357-8461-4e8af43a1d5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.775770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775751 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.775841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775777 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.775841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775793 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.775841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775809 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed628cdb-692c-4357-8461-4e8af43a1d5c-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.775841 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.775822 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.777019 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.776991 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-envoy" pod "ed628cdb-692c-4357-8461-4e8af43a1d5c" (UID: "ed628cdb-692c-4357-8461-4e8af43a1d5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:56:36.777114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.777019 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-podinfo" pod "ed628cdb-692c-4357-8461-4e8af43a1d5c" (UID: "ed628cdb-692c-4357-8461-4e8af43a1d5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:56:36.777190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.777171 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-token" pod "ed628cdb-692c-4357-8461-4e8af43a1d5c" (UID: "ed628cdb-692c-4357-8461-4e8af43a1d5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:36.777486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.777462 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-kube-api-access-8wgtk" pod "ed628cdb-692c-4357-8461-4e8af43a1d5c" (UID: "ed628cdb-692c-4357-8461-4e8af43a1d5c"). InnerVolumeSpecName "kube-api-access-8wgtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:36.787781 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.787759 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb"] Oct 30 13:56:36.799746 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.799727 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/premium-w2-bb86fd-kserve-669fd7db8d-zxlrb"] Oct 30 13:56:36.828875 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.828849 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv"] Oct 30 13:56:36.836950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.836927 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bb86fd/distinct-w2-bb86fd-kserve-cddb5dd8c-cxqmv"] Oct 30 13:56:36.876970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.876934 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.876970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.876961 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.876970 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.876974 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgtk\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-kube-api-access-8wgtk\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:36.877203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:36.876988 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed628cdb-692c-4357-8461-4e8af43a1d5c-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:37.124657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:37.124619 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:37.680497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:37.680466 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" Oct 30 13:56:37.680497 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:37.680486 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj" event={"ID":"ed628cdb-692c-4357-8461-4e8af43a1d5c","Type":"ContainerDied","Data":"c25c1d5f0a572ef3dbed6b402ea409d07f04364c19bdd6379cfb033951ae890b"} Oct 30 13:56:37.681077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:37.680586 5515 scope.go:122] "RemoveContainer" containerID="aa74bb58d202ab9521bf686b9c3cd7796248f0bd5761b2861fdb3165beef11d3" Oct 30 13:56:37.744904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:37.744871 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj"] Oct 30 13:56:37.758075 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:37.758046 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-kpzxj"] Oct 30 13:56:38.000230 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:38.000154 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" path="/var/lib/kubelet/pods/72bf3958-6eff-4b10-9d14-9ae43712b46d/volumes" Oct 30 13:56:38.000617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:38.000602 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" path="/var/lib/kubelet/pods/91f6d43e-ecc0-47cb-890e-c1958dc967f2/volumes" Oct 30 13:56:38.001016 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:38.001004 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" path="/var/lib/kubelet/pods/ed628cdb-692c-4357-8461-4e8af43a1d5c/volumes" Oct 30 13:56:38.909666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:38.909635 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl"] Oct 30 13:56:38.910080 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:38.909854 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" podUID="3282aba7-a1bd-498e-9013-f18c53c15574" containerName="maas-api" containerID="cri-o://84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27" gracePeriod=30 Oct 30 13:56:39.009301 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.009260 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:39.169871 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.169810 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:56:39.199772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.199743 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-metrics-certs\") pod \"3282aba7-a1bd-498e-9013-f18c53c15574\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " Oct 30 13:56:39.199955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.199800 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3282aba7-a1bd-498e-9013-f18c53c15574-kube-api-access-7vh7b\" (UniqueName: \"kubernetes.io/projected/3282aba7-a1bd-498e-9013-f18c53c15574-kube-api-access-7vh7b\") pod \"3282aba7-a1bd-498e-9013-f18c53c15574\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " Oct 30 13:56:39.199955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.199844 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls\") pod \"3282aba7-a1bd-498e-9013-f18c53c15574\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " Oct 30 13:56:39.199955 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.199907 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3282aba7-a1bd-498e-9013-f18c53c15574-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3282aba7-a1bd-498e-9013-f18c53c15574-openshift-service-ca\") pod \"3282aba7-a1bd-498e-9013-f18c53c15574\" (UID: \"3282aba7-a1bd-498e-9013-f18c53c15574\") " Oct 30 13:56:39.200419 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.200385 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3282aba7-a1bd-498e-9013-f18c53c15574-openshift-service-ca" pod "3282aba7-a1bd-498e-9013-f18c53c15574" (UID: "3282aba7-a1bd-498e-9013-f18c53c15574"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:39.202039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.202019 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls" pod "3282aba7-a1bd-498e-9013-f18c53c15574" (UID: "3282aba7-a1bd-498e-9013-f18c53c15574"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:56:39.202130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.202067 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-metrics-certs" pod "3282aba7-a1bd-498e-9013-f18c53c15574" (UID: "3282aba7-a1bd-498e-9013-f18c53c15574"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:56:39.202130 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.202077 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3282aba7-a1bd-498e-9013-f18c53c15574-kube-api-access-7vh7b" pod "3282aba7-a1bd-498e-9013-f18c53c15574" (UID: "3282aba7-a1bd-498e-9013-f18c53c15574"). InnerVolumeSpecName "kube-api-access-7vh7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:39.229303 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.229255 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:39.301793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.301758 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.301793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.301787 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vh7b\" (UniqueName: \"kubernetes.io/projected/3282aba7-a1bd-498e-9013-f18c53c15574-kube-api-access-7vh7b\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.301793 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.301797 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3282aba7-a1bd-498e-9013-f18c53c15574-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.302083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.301806 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3282aba7-a1bd-498e-9013-f18c53c15574-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:39.440209 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.440119 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:39.559836 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.559798 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" podUID="c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:56:39.696077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.695978 5515 generic.go:356] "Generic (PLEG): container finished" podID="3282aba7-a1bd-498e-9013-f18c53c15574" containerID="84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27" exitCode=0 Oct 30 13:56:39.696077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.696029 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" Oct 30 13:56:39.696077 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.696061 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" event={"ID":"3282aba7-a1bd-498e-9013-f18c53c15574","Type":"ContainerDied","Data":"84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27"} Oct 30 13:56:39.696423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.696116 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl" event={"ID":"3282aba7-a1bd-498e-9013-f18c53c15574","Type":"ContainerDied","Data":"50dd34fd2c857a37f6ef22d0d2899cd603a48f60b90fa0eb558fa1d11ea119f7"} Oct 30 13:56:39.696423 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.696140 5515 scope.go:122] "RemoveContainer" containerID="84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27" Oct 30 13:56:39.706150 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.706128 5515 scope.go:122] "RemoveContainer" containerID="84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27" Oct 30 13:56:39.706530 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:39.706494 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27\": container with ID starting with 84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27 not found: ID does not exist" containerID="84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27" Oct 30 13:56:39.706638 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.706534 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27"} err="failed to get container status \"84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27\": rpc error: code = NotFound desc = could not find container \"84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27\": container with ID starting with 84018c8c6e3e69135e51bb84bd078e91e90573a5405c3c9d3a12bbbabd0e7a27 not found: ID does not exist" Oct 30 13:56:39.750718 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.750686 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl"] Oct 30 13:56:39.754161 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.754135 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-241ffa-68d47c8696-pvksl"] Oct 30 13:56:39.766645 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.766615 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:39.992895 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:39.992809 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:40.001712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.001685 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3282aba7-a1bd-498e-9013-f18c53c15574" path="/var/lib/kubelet/pods/3282aba7-a1bd-498e-9013-f18c53c15574/volumes" Oct 30 13:56:40.408203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.408163 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:40.504186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.504121 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:56:40.617172 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.617143 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b7cce7-5160-459f-b487-a0c3074697cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b7cce7-5160-459f-b487-a0c3074697cd-plugins-config-volume\") pod \"60b7cce7-5160-459f-b487-a0c3074697cd\" (UID: \"60b7cce7-5160-459f-b487-a0c3074697cd\") " Oct 30 13:56:40.617366 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.617313 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b7cce7-5160-459f-b487-a0c3074697cd-kube-api-access-v4d8n\" (UniqueName: \"kubernetes.io/projected/60b7cce7-5160-459f-b487-a0c3074697cd-kube-api-access-v4d8n\") pod \"60b7cce7-5160-459f-b487-a0c3074697cd\" (UID: \"60b7cce7-5160-459f-b487-a0c3074697cd\") " Oct 30 13:56:40.617634 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.617599 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b7cce7-5160-459f-b487-a0c3074697cd-plugins-config-volume" pod "60b7cce7-5160-459f-b487-a0c3074697cd" (UID: "60b7cce7-5160-459f-b487-a0c3074697cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:40.619488 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.619467 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b7cce7-5160-459f-b487-a0c3074697cd-kube-api-access-v4d8n" pod "60b7cce7-5160-459f-b487-a0c3074697cd" (UID: "60b7cce7-5160-459f-b487-a0c3074697cd"). InnerVolumeSpecName "kube-api-access-v4d8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:40.704854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.704817 5515 generic.go:356] "Generic (PLEG): container finished" podID="60b7cce7-5160-459f-b487-a0c3074697cd" containerID="e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1" exitCode=0 Oct 30 13:56:40.705039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.704861 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" Oct 30 13:56:40.705039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.704865 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" event={"ID":"60b7cce7-5160-459f-b487-a0c3074697cd","Type":"ContainerDied","Data":"e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1"} Oct 30 13:56:40.705039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.704914 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8" event={"ID":"60b7cce7-5160-459f-b487-a0c3074697cd","Type":"ContainerDied","Data":"871c9a9df2a2e0ab8e99ac5369c7005ff5ba3486461343bec2ddd1f96e190c02"} Oct 30 13:56:40.705039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.704943 5515 scope.go:122] "RemoveContainer" containerID="e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1" Oct 30 13:56:40.714926 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.714905 5515 scope.go:122] "RemoveContainer" containerID="e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1" Oct 30 13:56:40.715224 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:40.715205 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1\": container with ID starting with e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1 not found: ID does not exist" containerID="e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1" Oct 30 13:56:40.715290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.715234 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1"} err="failed to get container status \"e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1\": rpc error: code = NotFound desc = could not find container \"e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1\": container with ID starting with e26af6ed955a8c7281c02657e39202a7190eebad1a2e19af3490976854e529c1 not found: ID does not exist" Oct 30 13:56:40.718131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.718109 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b7cce7-5160-459f-b487-a0c3074697cd-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:40.718220 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.718133 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4d8n\" (UniqueName: \"kubernetes.io/projected/60b7cce7-5160-459f-b487-a0c3074697cd-kube-api-access-v4d8n\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:40.756029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.755950 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8"] Oct 30 13:56:40.758799 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:40.758774 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-75d385-5d5cdd6f58-fbfm8"] Oct 30 13:56:41.693399 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:41.693370 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:42.000979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:42.000903 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b7cce7-5160-459f-b487-a0c3074697cd" path="/var/lib/kubelet/pods/60b7cce7-5160-459f-b487-a0c3074697cd/volumes" Oct 30 13:56:43.440403 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:43.440369 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:45.090649 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:45.090617 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:45.639391 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:45.639346 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" podUID="3d828ed9-4e9f-4194-a442-bd8f9059be68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:56:46.780570 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:46.780540 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:47.246725 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:47.246689 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:47.527580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:47.527502 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:47.742236 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:47.742199 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:47.944507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:47.944465 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:48.473510 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:48.473473 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:48.686367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:48.686337 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:48.913406 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:48.913371 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:49.465418 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:49.465381 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:49.559650 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:49.559607 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" podUID="c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:56:49.838580 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:49.838500 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:50.046742 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:50.046708 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:50.249139 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:50.249108 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:50.793641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:50.793601 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:51.016256 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.016204 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:51.448091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.448024 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 13:56:51.452849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.452824 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:51.541474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.541440 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-plugins-config-volume\") pod \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\" (UID: \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\") " Oct 30 13:56:51.541686 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.541515 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-kube-api-access-5xmhn\" (UniqueName: \"kubernetes.io/projected/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-kube-api-access-5xmhn\") pod \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\" (UID: \"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0\") " Oct 30 13:56:51.541931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.541884 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-plugins-config-volume" pod "c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" (UID: "c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:51.543833 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.543806 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-kube-api-access-5xmhn" pod "c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" (UID: "c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0"). InnerVolumeSpecName "kube-api-access-5xmhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:51.642676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.642633 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:51.642676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.642668 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmhn\" (UniqueName: \"kubernetes.io/projected/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0-kube-api-access-5xmhn\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:51.770103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.769963 5515 generic.go:356] "Generic (PLEG): container finished" podID="c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" containerID="035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b" exitCode=0 Oct 30 13:56:51.770103 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.770023 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" Oct 30 13:56:51.770385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.770105 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" event={"ID":"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0","Type":"ContainerDied","Data":"035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b"} Oct 30 13:56:51.770385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.770147 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg" event={"ID":"c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0","Type":"ContainerDied","Data":"bb7f3bbad300f87ee340767389cae7f6dae34f5bc3267fd53ebf22bb1692970f"} Oct 30 13:56:51.770385 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.770172 5515 scope.go:122] "RemoveContainer" containerID="035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b" Oct 30 13:56:51.780568 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.780543 5515 scope.go:122] "RemoveContainer" containerID="035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b" Oct 30 13:56:51.780864 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:51.780843 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b\": container with ID starting with 035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b not found: ID does not exist" containerID="035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b" Oct 30 13:56:51.780925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.780874 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b"} err="failed to get container status \"035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b\": rpc error: code = NotFound desc = could not find container \"035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b\": container with ID starting with 035b2083bceaf31385f89c5c0dfaf61a02d491e4301610f02ed866ab2c99ed1b not found: ID does not exist" Oct 30 13:56:51.822465 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.822437 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg"] Oct 30 13:56:51.827331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:51.827307 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-bb86fd-fb85fbf4b-vd9hg"] Oct 30 13:56:52.000304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:52.000255 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" path="/var/lib/kubelet/pods/c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0/volumes" Oct 30 13:56:52.691666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:52.691633 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:54.477162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:54.477113 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:54.700600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:54.700562 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:55.338788 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.338721 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:56:55.481706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.481671 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d828ed9-4e9f-4194-a442-bd8f9059be68-kube-api-access-78mzm\" (UniqueName: \"kubernetes.io/projected/3d828ed9-4e9f-4194-a442-bd8f9059be68-kube-api-access-78mzm\") pod \"3d828ed9-4e9f-4194-a442-bd8f9059be68\" (UID: \"3d828ed9-4e9f-4194-a442-bd8f9059be68\") " Oct 30 13:56:55.482199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.481834 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d828ed9-4e9f-4194-a442-bd8f9059be68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d828ed9-4e9f-4194-a442-bd8f9059be68-plugins-config-volume\") pod \"3d828ed9-4e9f-4194-a442-bd8f9059be68\" (UID: \"3d828ed9-4e9f-4194-a442-bd8f9059be68\") " Oct 30 13:56:55.482199 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.482170 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d828ed9-4e9f-4194-a442-bd8f9059be68-plugins-config-volume" pod "3d828ed9-4e9f-4194-a442-bd8f9059be68" (UID: "3d828ed9-4e9f-4194-a442-bd8f9059be68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:56:55.483735 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.483713 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d828ed9-4e9f-4194-a442-bd8f9059be68-kube-api-access-78mzm" pod "3d828ed9-4e9f-4194-a442-bd8f9059be68" (UID: "3d828ed9-4e9f-4194-a442-bd8f9059be68"). InnerVolumeSpecName "kube-api-access-78mzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:56:55.583327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.583296 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d828ed9-4e9f-4194-a442-bd8f9059be68-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:55.583327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.583323 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78mzm\" (UniqueName: \"kubernetes.io/projected/3d828ed9-4e9f-4194-a442-bd8f9059be68-kube-api-access-78mzm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:56:55.795248 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.795211 5515 generic.go:356] "Generic (PLEG): container finished" podID="3d828ed9-4e9f-4194-a442-bd8f9059be68" containerID="22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb" exitCode=0 Oct 30 13:56:55.795463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.795255 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" Oct 30 13:56:55.795463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.795257 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" event={"ID":"3d828ed9-4e9f-4194-a442-bd8f9059be68","Type":"ContainerDied","Data":"22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb"} Oct 30 13:56:55.795463 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.795438 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4" event={"ID":"3d828ed9-4e9f-4194-a442-bd8f9059be68","Type":"ContainerDied","Data":"17046ee6abd06af140d8730882b7a954821b5ef909e4f93fd756dff6788b29ab"} Oct 30 13:56:55.795654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.795469 5515 scope.go:122] "RemoveContainer" containerID="22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb" Oct 30 13:56:55.805304 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.805285 5515 scope.go:122] "RemoveContainer" containerID="22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb" Oct 30 13:56:55.805566 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:55.805546 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb\": container with ID starting with 22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb not found: ID does not exist" containerID="22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb" Oct 30 13:56:55.805637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.805573 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb"} err="failed to get container status \"22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb\": rpc error: code = NotFound desc = could not find container \"22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb\": container with ID starting with 22cdf10d16b304fddcf05df78e7c2a470e08b00a6b0c5adb14ee9cac833c0eeb not found: ID does not exist" Oct 30 13:56:55.846145 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.846113 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4"] Oct 30 13:56:55.850706 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:55.850683 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-241ffa-8ffc69466-rv2p4"] Oct 30 13:56:56.000214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:56.000181 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d828ed9-4e9f-4194-a442-bd8f9059be68" path="/var/lib/kubelet/pods/3d828ed9-4e9f-4194-a442-bd8f9059be68/volumes" Oct 30 13:56:56.089824 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:56.089721 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:56.758356 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:56.758327 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:56.965514 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:56.965479 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:57.710069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:57.710026 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:58.987213 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:56:58.987171 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:56:59.151480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:59.151443 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:56:59.156244 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:59.156212 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:59.406329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:59.406300 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:59.624386 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:56:59.624348 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:00.404674 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:00.404631 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l"] Oct 30 13:57:00.651438 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:00.651401 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:00.851709 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:00.851624 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:01.032505 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.032468 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk"] Oct 30 13:57:01.033212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033183 5515 cpu_manager.go:414] "containerMap: removing container" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" Oct 30 13:57:01.033212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033210 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033226 5515 cpu_manager.go:414] "containerMap: removing container" podUID="60b7cce7-5160-459f-b487-a0c3074697cd" containerName="payload-processing" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033234 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b7cce7-5160-459f-b487-a0c3074697cd" containerName="payload-processing" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033252 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" containerName="payload-processing" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033260 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" containerName="payload-processing" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033285 5515 cpu_manager.go:414] "containerMap: removing container" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033292 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033303 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3d828ed9-4e9f-4194-a442-bd8f9059be68" containerName="payload-processing" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033312 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d828ed9-4e9f-4194-a442-bd8f9059be68" containerName="payload-processing" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033339 5515 cpu_manager.go:414] "containerMap: removing container" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033345 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033362 5515 cpu_manager.go:414] "containerMap: removing container" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033369 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033382 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0bacc714-649c-4d1b-9ee5-fce729d9d7ef" containerName="maas-api" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033390 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bacc714-649c-4d1b-9ee5-fce729d9d7ef" containerName="maas-api" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033402 5515 cpu_manager.go:414] "containerMap: removing container" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033409 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033425 5515 cpu_manager.go:414] "containerMap: removing container" podUID="3282aba7-a1bd-498e-9013-f18c53c15574" containerName="maas-api" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033431 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3282aba7-a1bd-498e-9013-f18c53c15574" containerName="maas-api" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033452 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033460 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" Oct 30 13:57:01.033466 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033473 5515 cpu_manager.go:414] "containerMap: removing container" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033480 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033496 5515 cpu_manager.go:414] "containerMap: removing container" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033503 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033515 5515 cpu_manager.go:414] "containerMap: removing container" podUID="c89a6cc5-1098-4fbf-baa4-0d2f47200d28" containerName="revoke-keys" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033522 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89a6cc5-1098-4fbf-baa4-0d2f47200d28" containerName="revoke-keys" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033630 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b4ca2e-e76e-46b0-b514-a03d9827e132" containerName="istio-proxy" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033640 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033653 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed628cdb-692c-4357-8461-4e8af43a1d5c" containerName="istio-proxy" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033664 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033673 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bacc714-649c-4d1b-9ee5-fce729d9d7ef" containerName="maas-api" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033682 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb79059-60be-46d7-a8d5-249714d32aab" containerName="istio-proxy" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033692 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3282aba7-a1bd-498e-9013-f18c53c15574" containerName="maas-api" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033703 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033711 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f6d43e-ecc0-47cb-890e-c1958dc967f2" containerName="main" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033720 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0890391b-13c2-422d-90cf-3ac31659e601" containerName="istio-proxy" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033727 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89a6cc5-1098-4fbf-baa4-0d2f47200d28" containerName="revoke-keys" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033734 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bf3958-6eff-4b10-9d14-9ae43712b46d" containerName="main" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033742 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b7cce7-5160-459f-b487-a0c3074697cd" containerName="payload-processing" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033752 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b6b991-deb5-44d6-a8e2-25af9eb6e6b0" containerName="payload-processing" Oct 30 13:57:01.034687 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.033761 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d828ed9-4e9f-4194-a442-bd8f9059be68" containerName="payload-processing" Oct 30 13:57:01.038816 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.038791 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.067076 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.067044 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:57:01.067226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.067113 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\"" Oct 30 13:57:01.068915 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.068896 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk"] Oct 30 13:57:01.070168 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.070148 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:01.136212 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.136178 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr4m5\" (UniqueName: \"kubernetes.io/projected/d9b48c1f-cc31-4af2-8a47-bbae81462a22-kube-api-access-lr4m5\") pod \"maas-api-revoke-sub-26e4c3e4-nrxrk\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.136429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.136352 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26e4c3e4-nrxrk\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.136429 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.136399 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26e4c3e4-nrxrk\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.237770 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.237736 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26e4c3e4-nrxrk\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.237938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.237777 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26e4c3e4-nrxrk\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.237938 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.237856 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr4m5\" (UniqueName: \"kubernetes.io/projected/d9b48c1f-cc31-4af2-8a47-bbae81462a22-kube-api-access-lr4m5\") pod \"maas-api-revoke-sub-26e4c3e4-nrxrk\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.238576 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.238554 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26e4c3e4-nrxrk\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.240226 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.240203 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26e4c3e4-nrxrk\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.260744 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.260719 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr4m5\" (UniqueName: \"kubernetes.io/projected/d9b48c1f-cc31-4af2-8a47-bbae81462a22-kube-api-access-lr4m5\") pod \"maas-api-revoke-sub-26e4c3e4-nrxrk\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.354493 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.354458 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:01.521132 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.521099 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:01.658654 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.658629 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk"] Oct 30 13:57:01.659245 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:57:01.659193 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b48c1f_cc31_4af2_8a47_bbae81462a22.slice/crio-e80a931af21a8f1bd9e2ab4201013e5498f139dc88303d81ed6d868010fa5547 WatchSource:0}: Error finding container e80a931af21a8f1bd9e2ab4201013e5498f139dc88303d81ed6d868010fa5547: Status 404 returned error can't find the container with id e80a931af21a8f1bd9e2ab4201013e5498f139dc88303d81ed6d868010fa5547 Oct 30 13:57:01.829936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:01.829901 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" event={"ID":"d9b48c1f-cc31-4af2-8a47-bbae81462a22","Type":"ContainerStarted","Data":"e80a931af21a8f1bd9e2ab4201013e5498f139dc88303d81ed6d868010fa5547"} Oct 30 13:57:02.327186 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:02.327154 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:02.566922 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:02.566892 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:02.837643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:02.837605 5515 generic.go:356] "Generic (PLEG): container finished" podID="d9b48c1f-cc31-4af2-8a47-bbae81462a22" containerID="14be9ece6d113a8b66d2647d6669a544fe68988f8fcee748e22d94f5585de6a6" exitCode=0 Oct 30 13:57:02.837848 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:02.837689 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" event={"ID":"d9b48c1f-cc31-4af2-8a47-bbae81462a22","Type":"ContainerDied","Data":"14be9ece6d113a8b66d2647d6669a544fe68988f8fcee748e22d94f5585de6a6"} Oct 30 13:57:03.203410 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:03.203371 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:03.982231 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:03.982194 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:04.061330 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.061301 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-xks-ca\") pod \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " Oct 30 13:57:04.061494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.061346 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-openshift-ca\") pod \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " Oct 30 13:57:04.061494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.061403 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b48c1f-cc31-4af2-8a47-bbae81462a22-kube-api-access-lr4m5\" (UniqueName: \"kubernetes.io/projected/d9b48c1f-cc31-4af2-8a47-bbae81462a22-kube-api-access-lr4m5\") pod \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\" (UID: \"d9b48c1f-cc31-4af2-8a47-bbae81462a22\") " Oct 30 13:57:04.061789 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.061762 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-openshift-ca" pod "d9b48c1f-cc31-4af2-8a47-bbae81462a22" (UID: "d9b48c1f-cc31-4af2-8a47-bbae81462a22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:04.061872 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.061832 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-openshift-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:04.063392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.063367 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-xks-ca" pod "d9b48c1f-cc31-4af2-8a47-bbae81462a22" (UID: "d9b48c1f-cc31-4af2-8a47-bbae81462a22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:57:04.063526 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.063508 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b48c1f-cc31-4af2-8a47-bbae81462a22-kube-api-access-lr4m5" pod "d9b48c1f-cc31-4af2-8a47-bbae81462a22" (UID: "d9b48c1f-cc31-4af2-8a47-bbae81462a22"). InnerVolumeSpecName "kube-api-access-lr4m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:04.162763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.162737 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b48c1f-cc31-4af2-8a47-bbae81462a22-maas-api-xks-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:04.162763 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.162762 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr4m5\" (UniqueName: \"kubernetes.io/projected/d9b48c1f-cc31-4af2-8a47-bbae81462a22-kube-api-access-lr4m5\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:04.851162 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.851129 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" Oct 30 13:57:04.851291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.851164 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26e4c3e4-nrxrk" event={"ID":"d9b48c1f-cc31-4af2-8a47-bbae81462a22","Type":"ContainerDied","Data":"e80a931af21a8f1bd9e2ab4201013e5498f139dc88303d81ed6d868010fa5547"} Oct 30 13:57:04.851291 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.851227 5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e80a931af21a8f1bd9e2ab4201013e5498f139dc88303d81ed6d868010fa5547" Oct 30 13:57:04.895900 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:04.895865 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:05.168574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.168537 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:05.405860 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.405825 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" podUID="75668c46-4685-4d6e-8a91-cc8f21364998" containerName="payload-processing" containerID="cri-o://a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530" gracePeriod=25 Oct 30 13:57:05.657643 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.657619 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 13:57:05.676395 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.676330 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75668c46-4685-4d6e-8a91-cc8f21364998-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75668c46-4685-4d6e-8a91-cc8f21364998-plugins-config-volume\") pod \"75668c46-4685-4d6e-8a91-cc8f21364998\" (UID: \"75668c46-4685-4d6e-8a91-cc8f21364998\") " Oct 30 13:57:05.676503 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.676437 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75668c46-4685-4d6e-8a91-cc8f21364998-kube-api-access-5jdkk\" (UniqueName: \"kubernetes.io/projected/75668c46-4685-4d6e-8a91-cc8f21364998-kube-api-access-5jdkk\") pod \"75668c46-4685-4d6e-8a91-cc8f21364998\" (UID: \"75668c46-4685-4d6e-8a91-cc8f21364998\") " Oct 30 13:57:05.676760 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.676737 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75668c46-4685-4d6e-8a91-cc8f21364998-plugins-config-volume" pod "75668c46-4685-4d6e-8a91-cc8f21364998" (UID: "75668c46-4685-4d6e-8a91-cc8f21364998"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:05.676865 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.676849 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75668c46-4685-4d6e-8a91-cc8f21364998-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:05.678522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.678501 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75668c46-4685-4d6e-8a91-cc8f21364998-kube-api-access-5jdkk" pod "75668c46-4685-4d6e-8a91-cc8f21364998" (UID: "75668c46-4685-4d6e-8a91-cc8f21364998"). InnerVolumeSpecName "kube-api-access-5jdkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:05.777874 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.777846 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdkk\" (UniqueName: \"kubernetes.io/projected/75668c46-4685-4d6e-8a91-cc8f21364998-kube-api-access-5jdkk\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:05.858905 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.858873 5515 generic.go:356] "Generic (PLEG): container finished" podID="75668c46-4685-4d6e-8a91-cc8f21364998" containerID="a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530" exitCode=0 Oct 30 13:57:05.859087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.858926 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" Oct 30 13:57:05.859087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.858961 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" event={"ID":"75668c46-4685-4d6e-8a91-cc8f21364998","Type":"ContainerDied","Data":"a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530"} Oct 30 13:57:05.859087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.859017 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l" event={"ID":"75668c46-4685-4d6e-8a91-cc8f21364998","Type":"ContainerDied","Data":"b0845ad473bb7b7d95d2deb647f2110bd837828df4af1e4b803201816a466d13"} Oct 30 13:57:05.859087 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.859046 5515 scope.go:122] "RemoveContainer" containerID="a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530" Oct 30 13:57:05.868227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.868187 5515 scope.go:122] "RemoveContainer" containerID="a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530" Oct 30 13:57:05.868494 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:57:05.868476 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530\": container with ID starting with a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530 not found: ID does not exist" containerID="a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530" Oct 30 13:57:05.868569 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.868508 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530"} err="failed to get container status \"a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530\": rpc error: code = NotFound desc = could not find container \"a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530\": container with ID starting with a980e62c0d82c0ed9faebc2b00b8a2ba2178dfe8111520c0d0aaa6bd9366e530 not found: ID does not exist" Oct 30 13:57:05.915524 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.915491 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l"] Oct 30 13:57:05.919494 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:05.919470 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7a8b03-84f9f89544-dsz5l"] Oct 30 13:57:06.001023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:06.000947 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75668c46-4685-4d6e-8a91-cc8f21364998" path="/var/lib/kubelet/pods/75668c46-4685-4d6e-8a91-cc8f21364998/volumes" Oct 30 13:57:06.713912 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:06.713876 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:08.442522 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:08.442486 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:10.019312 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:10.019246 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:10.421203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:10.421171 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:10.511319 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:10.511282 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x"] Oct 30 13:57:10.545398 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:10.545363 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l"] Oct 30 13:57:10.833400 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:10.833318 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:11.244223 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:11.244193 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:11.674794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:11.674761 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:12.071371 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:12.071240 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:12.448125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:12.448087 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:12.833880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:12.833787 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:13.215857 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:13.215805 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:13.617506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:13.617395 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:14.084628 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:14.084595 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:14.476525 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:14.476496 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:15.274550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:15.274510 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:15.512217 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:15.512171 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" podUID="48085915-c753-4106-9bc8-faa198f108f9" containerName="payload-processing" containerID="cri-o://ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73" gracePeriod=25 Oct 30 13:57:15.546376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:15.546304 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" podUID="11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" containerName="payload-pre-processing" containerID="cri-o://03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce" gracePeriod=25 Oct 30 13:57:15.789183 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:15.789155 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:16.158451 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:16.158390 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:16.226950 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:16.226916 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:57:16.560680 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:16.560574 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:16.955804 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:16.955773 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:17.439582 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:17.439542 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:17.843561 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:17.843484 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:18.747827 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:18.747794 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" podUID="11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:57:18.748003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:18.747796 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" podUID="48085915-c753-4106-9bc8-faa198f108f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:57:19.509028 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:19.508999 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:21.222433 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:21.222402 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:21.789469 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:21.789436 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:22.163885 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:22.163842 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:22.629430 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:22.629344 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:23.011956 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:23.011920 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:23.947233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:23.947200 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:57:24.480455 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:24.480423 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:24.790611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:24.790522 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d"] Oct 30 13:57:24.790794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:24.790757 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" containerID="cri-o://662bbfeb975500dd20c78f781ba94f1371d4f24bd3621192523d81bffa2c306d" gracePeriod=30 Oct 30 13:57:24.856065 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:24.856022 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:26.445290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:26.445239 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:27.679819 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:27.679786 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl"] Oct 30 13:57:27.696290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:27.694768 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9"] Oct 30 13:57:27.822086 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:27.822055 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:28.196290 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:28.196234 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:28.747945 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:28.747907 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" podUID="48085915-c753-4106-9bc8-faa198f108f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:57:28.748402 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:28.747942 5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" podUID="11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:57:28.988475 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:57:28.988437 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:57:29.783614 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:29.783582 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:29.992807 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:29.992777 5515 generic.go:356] "Generic (PLEG): container finished" podID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerID="662bbfeb975500dd20c78f781ba94f1371d4f24bd3621192523d81bffa2c306d" exitCode=0 Oct 30 13:57:29.992964 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:29.992853 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" event={"ID":"b0abe68e-ab48-41a6-bcdb-ab612596352b","Type":"ContainerDied","Data":"662bbfeb975500dd20c78f781ba94f1371d4f24bd3621192523d81bffa2c306d"} Oct 30 13:57:30.054512 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.054489 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 13:57:30.130904 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.130873 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-kube-api-access-8p7r2\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-kube-api-access-8p7r2\") pod \"b0abe68e-ab48-41a6-bcdb-ab612596352b\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " Oct 30 13:57:30.131091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.130934 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-podinfo\") pod \"b0abe68e-ab48-41a6-bcdb-ab612596352b\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " Oct 30 13:57:30.131091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.130986 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-envoy\") pod \"b0abe68e-ab48-41a6-bcdb-ab612596352b\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " Oct 30 13:57:30.131091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131008 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0abe68e-ab48-41a6-bcdb-ab612596352b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe68e-ab48-41a6-bcdb-ab612596352b-istiod-ca-cert\") pod \"b0abe68e-ab48-41a6-bcdb-ab612596352b\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " Oct 30 13:57:30.131091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131037 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-socket\") pod \"b0abe68e-ab48-41a6-bcdb-ab612596352b\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " Oct 30 13:57:30.131091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131058 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-credential-socket\") pod \"b0abe68e-ab48-41a6-bcdb-ab612596352b\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " Oct 30 13:57:30.131091 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131088 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-token\") pod \"b0abe68e-ab48-41a6-bcdb-ab612596352b\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " Oct 30 13:57:30.131501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131123 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-certs\") pod \"b0abe68e-ab48-41a6-bcdb-ab612596352b\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " Oct 30 13:57:30.131501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131166 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-data\") pod \"b0abe68e-ab48-41a6-bcdb-ab612596352b\" (UID: \"b0abe68e-ab48-41a6-bcdb-ab612596352b\") " Oct 30 13:57:30.131501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131359 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-socket" pod "b0abe68e-ab48-41a6-bcdb-ab612596352b" (UID: "b0abe68e-ab48-41a6-bcdb-ab612596352b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:30.131501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131374 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-credential-socket" pod "b0abe68e-ab48-41a6-bcdb-ab612596352b" (UID: "b0abe68e-ab48-41a6-bcdb-ab612596352b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:30.131740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131655 5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:30.131740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131677 5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-credential-socket\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:30.131896 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131874 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-certs" pod "b0abe68e-ab48-41a6-bcdb-ab612596352b" (UID: "b0abe68e-ab48-41a6-bcdb-ab612596352b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:30.131976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131893 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0abe68e-ab48-41a6-bcdb-ab612596352b-istiod-ca-cert" pod "b0abe68e-ab48-41a6-bcdb-ab612596352b" (UID: "b0abe68e-ab48-41a6-bcdb-ab612596352b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:30.131976 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.131956 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-data" pod "b0abe68e-ab48-41a6-bcdb-ab612596352b" (UID: "b0abe68e-ab48-41a6-bcdb-ab612596352b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:30.133340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.133310 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-envoy" pod "b0abe68e-ab48-41a6-bcdb-ab612596352b" (UID: "b0abe68e-ab48-41a6-bcdb-ab612596352b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:30.133461 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.133443 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-token" pod "b0abe68e-ab48-41a6-bcdb-ab612596352b" (UID: "b0abe68e-ab48-41a6-bcdb-ab612596352b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:30.133562 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.133547 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-podinfo" pod "b0abe68e-ab48-41a6-bcdb-ab612596352b" (UID: "b0abe68e-ab48-41a6-bcdb-ab612596352b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:57:30.133696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.133677 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-kube-api-access-8p7r2" pod "b0abe68e-ab48-41a6-bcdb-ab612596352b" (UID: "b0abe68e-ab48-41a6-bcdb-ab612596352b"). InnerVolumeSpecName "kube-api-access-8p7r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:30.232506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.232477 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p7r2\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-kube-api-access-8p7r2\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:30.232506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.232502 5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-podinfo\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:30.232506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.232512 5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-envoy\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:30.232707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.232520 5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0abe68e-ab48-41a6-bcdb-ab612596352b-istiod-ca-cert\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:30.232707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.232528 5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-token\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:30.232707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.232535 5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-workload-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:30.232707 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.232542 5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0abe68e-ab48-41a6-bcdb-ab612596352b-istio-data\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:30.687082 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.687033 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:31.000131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.999978 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" Oct 30 13:57:31.000131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:30.999976 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d" event={"ID":"b0abe68e-ab48-41a6-bcdb-ab612596352b","Type":"ContainerDied","Data":"d00340fdbe9fbec55ae47cbf1f06dc6d04e068f957b5c46b09f69f671d499d5f"} Oct 30 13:57:31.013202 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:31.000135 5515 scope.go:122] "RemoveContainer" containerID="662bbfeb975500dd20c78f781ba94f1371d4f24bd3621192523d81bffa2c306d" Oct 30 13:57:31.065989 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:31.065960 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d"] Oct 30 13:57:31.078136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:31.078114 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-tw62d"] Oct 30 13:57:31.158255 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:31.158224 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:31.533667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:31.533631 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:31.999600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:31.999568 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" path="/var/lib/kubelet/pods/b0abe68e-ab48-41a6-bcdb-ab612596352b/volumes" Oct 30 13:57:33.123811 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:33.123778 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:33.332701 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:33.332669 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5"] Oct 30 13:57:33.523457 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:33.523421 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:33.888427 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:33.888397 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:34.264554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:34.264432 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:34.807925 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:34.807862 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:35.795543 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.795519 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:57:35.819859 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.819838 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:57:35.890444 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.890414 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48085915-c753-4106-9bc8-faa198f108f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48085915-c753-4106-9bc8-faa198f108f9-plugins-config-volume\") pod \"48085915-c753-4106-9bc8-faa198f108f9\" (UID: \"48085915-c753-4106-9bc8-faa198f108f9\") " Oct 30 13:57:35.890622 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.890507 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48085915-c753-4106-9bc8-faa198f108f9-kube-api-access-rlh2g\" (UniqueName: \"kubernetes.io/projected/48085915-c753-4106-9bc8-faa198f108f9-kube-api-access-rlh2g\") pod \"48085915-c753-4106-9bc8-faa198f108f9\" (UID: \"48085915-c753-4106-9bc8-faa198f108f9\") " Oct 30 13:57:35.890851 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.890826 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48085915-c753-4106-9bc8-faa198f108f9-plugins-config-volume" pod "48085915-c753-4106-9bc8-faa198f108f9" (UID: "48085915-c753-4106-9bc8-faa198f108f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:35.892586 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.892566 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48085915-c753-4106-9bc8-faa198f108f9-kube-api-access-rlh2g" pod "48085915-c753-4106-9bc8-faa198f108f9" (UID: "48085915-c753-4106-9bc8-faa198f108f9"). InnerVolumeSpecName "kube-api-access-rlh2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:35.992023 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.991994 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-kube-api-access-n7zcl\" (UniqueName: \"kubernetes.io/projected/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-kube-api-access-n7zcl\") pod \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\" (UID: \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\") " Oct 30 13:57:35.992159 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.992095 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-plugins-config-volume\") pod \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\" (UID: \"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575\") " Oct 30 13:57:35.992344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.992330 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlh2g\" (UniqueName: \"kubernetes.io/projected/48085915-c753-4106-9bc8-faa198f108f9-kube-api-access-rlh2g\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:35.992344 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.992345 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48085915-c753-4106-9bc8-faa198f108f9-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:35.992536 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.992515 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-plugins-config-volume" pod "11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" (UID: "11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:35.993985 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:35.993965 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-kube-api-access-n7zcl" pod "11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" (UID: "11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575"). InnerVolumeSpecName "kube-api-access-n7zcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:36.025907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.025843 5515 generic.go:356] "Generic (PLEG): container finished" podID="48085915-c753-4106-9bc8-faa198f108f9" containerID="ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73" exitCode=0 Oct 30 13:57:36.025907 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.025886 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" Oct 30 13:57:36.026069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.025918 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" event={"ID":"48085915-c753-4106-9bc8-faa198f108f9","Type":"ContainerDied","Data":"ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73"} Oct 30 13:57:36.026069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.025963 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x" event={"ID":"48085915-c753-4106-9bc8-faa198f108f9","Type":"ContainerDied","Data":"484fadb5ce046a7b8434f84a9f21b85a136d3dcac7a43832efa1b440bc718d1b"} Oct 30 13:57:36.026069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.025984 5515 scope.go:122] "RemoveContainer" containerID="ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73" Oct 30 13:57:36.027567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.027544 5515 generic.go:356] "Generic (PLEG): container finished" podID="11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" containerID="03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce" exitCode=0 Oct 30 13:57:36.027715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.027596 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" Oct 30 13:57:36.027715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.027568 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" event={"ID":"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575","Type":"ContainerDied","Data":"03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce"} Oct 30 13:57:36.027715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.027679 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l" event={"ID":"11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575","Type":"ContainerDied","Data":"28c4589fa9a93d5d6dab76e9c5c56110b08064523b99c6f2acb134a7698291b9"} Oct 30 13:57:36.035507 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.035491 5515 scope.go:122] "RemoveContainer" containerID="ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73" Oct 30 13:57:36.035775 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:57:36.035755 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73\": container with ID starting with ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73 not found: ID does not exist" containerID="ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73" Oct 30 13:57:36.035849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.035783 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73"} err="failed to get container status \"ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73\": rpc error: code = NotFound desc = could not find container \"ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73\": container with ID starting with ddbf6e4bcc34b2a8e9bf68a88245c76ba433577e934c993f418314bb8eb2cf73 not found: ID does not exist" Oct 30 13:57:36.035849 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.035799 5515 scope.go:122] "RemoveContainer" containerID="03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce" Oct 30 13:57:36.043845 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.043826 5515 scope.go:122] "RemoveContainer" containerID="03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce" Oct 30 13:57:36.044089 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:57:36.044072 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce\": container with ID starting with 03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce not found: ID does not exist" containerID="03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce" Oct 30 13:57:36.044153 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.044094 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce"} err="failed to get container status \"03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce\": rpc error: code = NotFound desc = could not find container \"03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce\": container with ID starting with 03eacb0cc571b0286cc275ea68943ac6ba16123e8578b22c99122502e092f2ce not found: ID does not exist" Oct 30 13:57:36.078296 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.078257 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l"] Oct 30 13:57:36.084999 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.084975 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-451a21-d9dd5f59f-qd57l"] Oct 30 13:57:36.093822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.093802 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:36.093822 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.093823 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zcl\" (UniqueName: \"kubernetes.io/projected/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575-kube-api-access-n7zcl\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:36.120195 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.120173 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x"] Oct 30 13:57:36.125772 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:36.125744 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-451a21-7f55f6f596-jpx5x"] Oct 30 13:57:37.591294 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:37.591236 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:38.000567 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.000534 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" path="/var/lib/kubelet/pods/11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575/volumes" Oct 30 13:57:38.000931 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.000918 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48085915-c753-4106-9bc8-faa198f108f9" path="/var/lib/kubelet/pods/48085915-c753-4106-9bc8-faa198f108f9/volumes" Oct 30 13:57:38.333396 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.333296 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" podUID="58a3c214-d603-45e7-8011-76b3e5d8bc67" containerName="payload-processing" containerID="cri-o://8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61" gracePeriod=25 Oct 30 13:57:38.594092 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.594035 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 13:57:38.615447 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.615423 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a3c214-d603-45e7-8011-76b3e5d8bc67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a3c214-d603-45e7-8011-76b3e5d8bc67-plugins-config-volume\") pod \"58a3c214-d603-45e7-8011-76b3e5d8bc67\" (UID: \"58a3c214-d603-45e7-8011-76b3e5d8bc67\") " Oct 30 13:57:38.615602 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.615519 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a3c214-d603-45e7-8011-76b3e5d8bc67-kube-api-access-nd2cr\" (UniqueName: \"kubernetes.io/projected/58a3c214-d603-45e7-8011-76b3e5d8bc67-kube-api-access-nd2cr\") pod \"58a3c214-d603-45e7-8011-76b3e5d8bc67\" (UID: \"58a3c214-d603-45e7-8011-76b3e5d8bc67\") " Oct 30 13:57:38.615850 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.615825 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a3c214-d603-45e7-8011-76b3e5d8bc67-plugins-config-volume" pod "58a3c214-d603-45e7-8011-76b3e5d8bc67" (UID: "58a3c214-d603-45e7-8011-76b3e5d8bc67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:38.617620 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.617598 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a3c214-d603-45e7-8011-76b3e5d8bc67-kube-api-access-nd2cr" pod "58a3c214-d603-45e7-8011-76b3e5d8bc67" (UID: "58a3c214-d603-45e7-8011-76b3e5d8bc67"). InnerVolumeSpecName "kube-api-access-nd2cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:38.716358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.716333 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2cr\" (UniqueName: \"kubernetes.io/projected/58a3c214-d603-45e7-8011-76b3e5d8bc67-kube-api-access-nd2cr\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:38.716358 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:38.716356 5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a3c214-d603-45e7-8011-76b3e5d8bc67-plugins-config-volume\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:39.048198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:39.048160 5515 generic.go:356] "Generic (PLEG): container finished" podID="58a3c214-d603-45e7-8011-76b3e5d8bc67" containerID="8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61" exitCode=0 Oct 30 13:57:39.048411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:39.048220 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" Oct 30 13:57:39.048411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:39.048239 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" event={"ID":"58a3c214-d603-45e7-8011-76b3e5d8bc67","Type":"ContainerDied","Data":"8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61"} Oct 30 13:57:39.048411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:39.048300 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5" event={"ID":"58a3c214-d603-45e7-8011-76b3e5d8bc67","Type":"ContainerDied","Data":"ba1395513269b92b194a07a5367d5c807aaf0969c73c4a5810002345d659a535"} Oct 30 13:57:39.048411 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:39.048324 5515 scope.go:122] "RemoveContainer" containerID="8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61" Oct 30 13:57:39.057565 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:39.057549 5515 scope.go:122] "RemoveContainer" containerID="8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61" Oct 30 13:57:39.057784 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:57:39.057768 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61\": container with ID starting with 8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61 not found: ID does not exist" containerID="8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61" Oct 30 13:57:39.057830 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:39.057793 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61"} err="failed to get container status \"8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61\": rpc error: code = NotFound desc = could not find container \"8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61\": container with ID starting with 8f77c39cd329626ce8c871b766234bf11eb5d65ccb09eaa0f645142da71d4c61 not found: ID does not exist" Oct 30 13:57:39.099621 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:39.099596 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5"] Oct 30 13:57:39.111935 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:39.111911 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-551dd1-5c7d8d7f44-2jnp5"] Oct 30 13:57:40.001004 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:40.000969 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a3c214-d603-45e7-8011-76b3e5d8bc67" path="/var/lib/kubelet/pods/58a3c214-d603-45e7-8011-76b3e5d8bc67/volumes" Oct 30 13:57:42.702020 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:42.701965 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" containerID="cri-o://67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5" gracePeriod=45 Oct 30 13:57:42.729696 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:42.729667 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" containerID="cri-o://96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6" gracePeriod=45 Oct 30 13:57:42.977882 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:42.977859 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 13:57:42.999203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:42.999179 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 13:57:43.054600 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.054567 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-model-cache\") pod \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " Oct 30 13:57:43.054785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.054657 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tls-certs\") pod \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " Oct 30 13:57:43.054785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.054700 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tmp-dir\") pod \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " Oct 30 13:57:43.054785 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.054731 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-model-cache\") pod \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " Oct 30 13:57:43.055000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.054846 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-model-cache" pod "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" (UID: "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:43.055000 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.054962 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-home\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-home\") pod \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " Oct 30 13:57:43.055129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055024 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-home\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-home\") pod \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " Oct 30 13:57:43.055129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055059 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-dshm\") pod \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " Oct 30 13:57:43.055129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.054962 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tmp-dir" pod "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" (UID: "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:43.055129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055114 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tmp-dir\") pod \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " Oct 30 13:57:43.055129 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.054979 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-model-cache" pod "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" (UID: "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:43.055443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055150 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-dshm\") pod \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " Oct 30 13:57:43.055443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055204 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-kube-api-access-htz85\" (UniqueName: \"kubernetes.io/projected/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-kube-api-access-htz85\") pod \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\" (UID: \"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1\") " Oct 30 13:57:43.055443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055233 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-home" pod "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" (UID: "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:43.055443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055244 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-kube-api-access-rhd4x\" (UniqueName: \"kubernetes.io/projected/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-kube-api-access-rhd4x\") pod \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " Oct 30 13:57:43.055443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055307 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tls-certs\") pod \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\" (UID: \"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859\") " Oct 30 13:57:43.055443 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055406 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-home" pod "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" (UID: "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:43.055784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055681 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.055784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055703 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.055784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055715 5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-model-cache\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.055784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055727 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.055784 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055738 5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-home\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.056138 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.055970 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tmp-dir" pod "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" (UID: "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:43.057234 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.057205 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tls-certs" pod "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" (UID: "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:57:43.058205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.058059 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-kube-api-access-htz85" pod "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" (UID: "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1"). InnerVolumeSpecName "kube-api-access-htz85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:43.058205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.058175 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tls-certs" pod "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" (UID: "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:57:43.058362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.058296 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-dshm" pod "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" (UID: "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:43.059471 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.059451 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-kube-api-access-rhd4x" pod "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" (UID: "f9aa5bf2-ad4f-4bca-b301-8c9f1812b859"). InnerVolumeSpecName "kube-api-access-rhd4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:43.059471 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.059457 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-dshm" pod "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" (UID: "1c2516ca-0565-4a71-ba56-48e1b1aa3fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:57:43.074355 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.074322 5515 generic.go:356] "Generic (PLEG): container finished" podID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerID="67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5" exitCode=0 Oct 30 13:57:43.074509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.074409 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" Oct 30 13:57:43.074509 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.074472 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" event={"ID":"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859","Type":"ContainerDied","Data":"67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5"} Oct 30 13:57:43.074630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.074509 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl" event={"ID":"f9aa5bf2-ad4f-4bca-b301-8c9f1812b859","Type":"ContainerDied","Data":"e027045c4a8b2aca7f135413388d4d2995b2e6aa224c92e8876d7c9ff8ff05e0"} Oct 30 13:57:43.074630 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.074530 5515 scope.go:122] "RemoveContainer" containerID="67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5" Oct 30 13:57:43.076721 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.076664 5515 generic.go:356] "Generic (PLEG): container finished" podID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerID="96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6" exitCode=0 Oct 30 13:57:43.076825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.076759 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" event={"ID":"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1","Type":"ContainerDied","Data":"96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6"} Oct 30 13:57:43.076825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.076789 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" event={"ID":"1c2516ca-0565-4a71-ba56-48e1b1aa3fc1","Type":"ContainerDied","Data":"7d445b2518beaa293b54b87cfacbf4098a5fcfd5a9fc58949415bf3e0c85aae4"} Oct 30 13:57:43.076954 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.076851 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9" Oct 30 13:57:43.086787 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.086759 5515 scope.go:122] "RemoveContainer" containerID="67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5" Oct 30 13:57:43.087115 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:57:43.087095 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5\": container with ID starting with 67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5 not found: ID does not exist" containerID="67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5" Oct 30 13:57:43.087184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.087125 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5"} err="failed to get container status \"67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5\": rpc error: code = NotFound desc = could not find container \"67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5\": container with ID starting with 67bfda31e2a7fbe634afc1a184ae6364ef7d9109dfbedc00d7287af33d7d1cd5 not found: ID does not exist" Oct 30 13:57:43.087184 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.087142 5515 scope.go:122] "RemoveContainer" containerID="96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6" Oct 30 13:57:43.097375 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.097356 5515 scope.go:122] "RemoveContainer" containerID="96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6" Oct 30 13:57:43.097703 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:57:43.097677 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6\": container with ID starting with 96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6 not found: ID does not exist" containerID="96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6" Oct 30 13:57:43.097782 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.097714 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6"} err="failed to get container status \"96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6\": rpc error: code = NotFound desc = could not find container \"96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6\": container with ID starting with 96ef08aff1490956d8466cbc4344dd0b32a633fc2a4f16771b29325cd39814b6 not found: ID does not exist" Oct 30 13:57:43.143550 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.143518 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl"] Oct 30 13:57:43.147205 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.147176 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/distinct-2-w3-551dd1-kserve-7dff4b96cc-rm5kl"] Oct 30 13:57:43.156554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.156531 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.156554 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.156553 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.156702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.156561 5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-tmp-dir\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.156702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.156569 5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-dshm\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.156702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.156578 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htz85\" (UniqueName: \"kubernetes.io/projected/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1-kube-api-access-htz85\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.156702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.156588 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhd4x\" (UniqueName: \"kubernetes.io/projected/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-kube-api-access-rhd4x\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.156702 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.156599 5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859-tls-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:43.183506 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.183479 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9"] Oct 30 13:57:43.184979 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.184960 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-551dd1/premium-w3-551dd1-kserve-86b4965b75-tjms9"] Oct 30 13:57:43.999324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.999243 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" path="/var/lib/kubelet/pods/1c2516ca-0565-4a71-ba56-48e1b1aa3fc1/volumes" Oct 30 13:57:43.999690 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:43.999677 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" path="/var/lib/kubelet/pods/f9aa5bf2-ad4f-4bca-b301-8c9f1812b859/volumes" Oct 30 13:57:49.329233 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:49.329196 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:57:51.387936 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.387893 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd"] Oct 30 13:57:51.388474 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.388139 5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" podUID="9870674e-3d75-4501-a5ce-4040294edd31" containerName="maas-api" containerID="cri-o://98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522" gracePeriod=30 Oct 30 13:57:51.483295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.483251 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:51.634107 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.634074 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 13:57:51.745102 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.745024 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-metrics-certs\") pod \"9870674e-3d75-4501-a5ce-4040294edd31\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " Oct 30 13:57:51.745313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.745147 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-maas-api-tls\") pod \"9870674e-3d75-4501-a5ce-4040294edd31\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " Oct 30 13:57:51.745313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.745172 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9870674e-3d75-4501-a5ce-4040294edd31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9870674e-3d75-4501-a5ce-4040294edd31-openshift-service-ca\") pod \"9870674e-3d75-4501-a5ce-4040294edd31\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " Oct 30 13:57:51.745313 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.745203 5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9870674e-3d75-4501-a5ce-4040294edd31-kube-api-access-6w49b\" (UniqueName: \"kubernetes.io/projected/9870674e-3d75-4501-a5ce-4040294edd31-kube-api-access-6w49b\") pod \"9870674e-3d75-4501-a5ce-4040294edd31\" (UID: \"9870674e-3d75-4501-a5ce-4040294edd31\") " Oct 30 13:57:51.745627 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.745593 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9870674e-3d75-4501-a5ce-4040294edd31-openshift-service-ca" pod "9870674e-3d75-4501-a5ce-4040294edd31" (UID: "9870674e-3d75-4501-a5ce-4040294edd31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:51.747316 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.747286 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-maas-api-tls" pod "9870674e-3d75-4501-a5ce-4040294edd31" (UID: "9870674e-3d75-4501-a5ce-4040294edd31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:57:51.747446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.747347 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9870674e-3d75-4501-a5ce-4040294edd31-kube-api-access-6w49b" pod "9870674e-3d75-4501-a5ce-4040294edd31" (UID: "9870674e-3d75-4501-a5ce-4040294edd31"). InnerVolumeSpecName "kube-api-access-6w49b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:51.747446 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.747396 5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-metrics-certs" pod "9870674e-3d75-4501-a5ce-4040294edd31" (UID: "9870674e-3d75-4501-a5ce-4040294edd31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:57:51.847003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.846965 5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-maas-api-tls\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:51.847003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.846992 5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9870674e-3d75-4501-a5ce-4040294edd31-openshift-service-ca\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:51.847003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.847003 5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w49b\" (UniqueName: \"kubernetes.io/projected/9870674e-3d75-4501-a5ce-4040294edd31-kube-api-access-6w49b\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:51.847003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:51.847012 5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9870674e-3d75-4501-a5ce-4040294edd31-metrics-certs\") on node \"ip-10-0-134-193.ec2.internal\" DevicePath \"\"" Oct 30 13:57:52.015467 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.015433 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:52.121381 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.121344 5515 generic.go:356] "Generic (PLEG): container finished" podID="9870674e-3d75-4501-a5ce-4040294edd31" containerID="98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522" exitCode=0 Oct 30 13:57:52.121584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.121390 5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" Oct 30 13:57:52.121584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.121384 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" event={"ID":"9870674e-3d75-4501-a5ce-4040294edd31","Type":"ContainerDied","Data":"98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522"} Oct 30 13:57:52.121584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.121508 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd" event={"ID":"9870674e-3d75-4501-a5ce-4040294edd31","Type":"ContainerDied","Data":"ba79f542136bd9739e218999c41aeb4357dafc8c0ecc4bced7a4f503b35c5863"} Oct 30 13:57:52.121584 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.121538 5515 scope.go:122] "RemoveContainer" containerID="98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522" Oct 30 13:57:52.130426 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.130408 5515 scope.go:122] "RemoveContainer" containerID="98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522" Oct 30 13:57:52.130676 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:57:52.130657 5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522\": container with ID starting with 98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522 not found: ID does not exist" containerID="98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522" Oct 30 13:57:52.130734 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.130684 5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522"} err="failed to get container status \"98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522\": rpc error: code = NotFound desc = could not find container \"98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522\": container with ID starting with 98c7f4908015d99e3eaade8d1873a2735b4ed3c9edb6d01dff0dea71ed4d7522 not found: ID does not exist" Oct 30 13:57:52.174676 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.174647 5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd"] Oct 30 13:57:52.187227 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.187196 5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-551dd1-5c875c6d7f-g2fbd"] Oct 30 13:57:52.420297 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.420252 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:52.899485 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:52.899452 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:54.001941 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:54.001906 5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9870674e-3d75-4501-a5ce-4040294edd31" path="/var/lib/kubelet/pods/9870674e-3d75-4501-a5ce-4040294edd31/volumes" Oct 30 13:57:58.757029 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:58.756997 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:58.986637 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:57:58.986604 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:57:59.124501 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:59.124409 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:59.530024 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:57:59.529986 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bd2wm"] Oct 30 13:58:00.585946 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:00.585919 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-77jxw_3abefab9-6297-457f-b668-3282fd99560b/manager/0.log" Oct 30 13:58:01.391733 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:01.391706 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bd2wm_10a1b17f-494a-432f-910c-7c8ac33ad3ad/limitador/0.log" Oct 30 13:58:02.398475 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:02.398446 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8489fb66bd-4jwwm_a1dcad86-b55c-40ed-9280-a639b93d15d4/manager/0.log" Oct 30 13:58:02.870477 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:02.870411 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gr5gq_3e186430-fae4-49df-a08b-29ab2912ecd2/manager/0.log" Oct 30 13:58:04.315487 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:04.315449 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:04.685825 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:04.685778 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:05.659639 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:05.659602 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:11.370726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:11.370684 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:15.783767 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:15.783732 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8489fb66bd-4jwwm_a1dcad86-b55c-40ed-9280-a639b93d15d4/manager/0.log" Oct 30 13:58:16.224589 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:16.224560 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gr5gq_3e186430-fae4-49df-a08b-29ab2912ecd2/manager/0.log" Oct 30 13:58:17.573611 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:17.573571 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:19.439759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:19.439717 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-26e4c3e4-nrxrk_d9b48c1f-cc31-4af2-8a47-bbae81462a22/revoke-keys/0.log" Oct 30 13:58:20.587006 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:20.586973 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-910539a4-frwtp_c89a6cc5-1098-4fbf-baa4-0d2f47200d28/revoke-keys/0.log" Oct 30 13:58:21.936093 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:21.936062 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-f69mt_792621b1-c1c3-4aa5-b5cf-0b07370bd2e4/postgres/0.log" Oct 30 13:58:23.963712 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:23.963684 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-77jxw_3abefab9-6297-457f-b668-3282fd99560b/manager/0.log" Oct 30 13:58:24.734224 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:24.734194 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bd2wm_10a1b17f-494a-432f-910c-7c8ac33ad3ad/limitador/0.log" Oct 30 13:58:26.360456 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:26.360428 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59688578c6-wtg9m_b98339fa-aad1-4672-9cc8-5c09532a01e2/kube-auth-proxy/0.log" Oct 30 13:58:26.750125 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:26.750093 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-ljfbp_2876231a-a44b-4a8f-b462-4d30201f90d5/istio-proxy/0.log" Oct 30 13:58:26.954034 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:26.954006 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-q8crv_e8370493-9181-48ef-bd2e-b3eab3293f98/istio-proxy/0.log" Oct 30 13:58:27.776249 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:27.776217 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-ld25h_b4951dbc-6cb4-40ce-af70-a8d331e7c2b8/payload-pre-processing/0.log" Oct 30 13:58:27.974694 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:27.974666 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bvptt_7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca/payload-processing/0.log" Oct 30 13:58:28.986860 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:58:28.986827 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:58:29.156013 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:29.155988 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9rg6h_6bac6b94-7947-41ce-aeb6-3daf8d972487/main/0.log" Oct 30 13:58:29.362059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:29.361990 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-pnkfq_b2ea78e5-dd16-43b8-a556-4fea83e22c64/main/0.log" Oct 30 13:58:29.927745 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:29.927716 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h57p9_d7d17f60-c53a-4058-a144-88472194f6e5/main/0.log" Oct 30 13:58:30.347879 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:30.347811 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-l9js2_ba929e02-50ce-44cf-af12-21d91c229651/main/0.log" Oct 30 13:58:30.543155 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:30.543126 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9zzbc_543202e9-3b15-46b2-9dad-b6d056c087b7/main/0.log" Oct 30 13:58:40.977906 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.977871 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z45hd/must-gather-2z97x"] Oct 30 13:58:40.978327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978256 5515 cpu_manager.go:414] "containerMap: removing container" podUID="58a3c214-d603-45e7-8011-76b3e5d8bc67" containerName="payload-processing" Oct 30 13:58:40.978327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978280 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a3c214-d603-45e7-8011-76b3e5d8bc67" containerName="payload-processing" Oct 30 13:58:40.978327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978297 5515 cpu_manager.go:414] "containerMap: removing container" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" Oct 30 13:58:40.978327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978302 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" Oct 30 13:58:40.978327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978310 5515 cpu_manager.go:414] "containerMap: removing container" podUID="48085915-c753-4106-9bc8-faa198f108f9" containerName="payload-processing" Oct 30 13:58:40.978327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978315 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48085915-c753-4106-9bc8-faa198f108f9" containerName="payload-processing" Oct 30 13:58:40.978327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978326 5515 cpu_manager.go:414] "containerMap: removing container" podUID="11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" containerName="payload-pre-processing" Oct 30 13:58:40.978327 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978331 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" containerName="payload-pre-processing" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978341 5515 cpu_manager.go:414] "containerMap: removing container" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978345 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978360 5515 cpu_manager.go:414] "containerMap: removing container" podUID="75668c46-4685-4d6e-8a91-cc8f21364998" containerName="payload-processing" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978365 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75668c46-4685-4d6e-8a91-cc8f21364998" containerName="payload-processing" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978375 5515 cpu_manager.go:414] "containerMap: removing container" podUID="d9b48c1f-cc31-4af2-8a47-bbae81462a22" containerName="revoke-keys" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978380 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b48c1f-cc31-4af2-8a47-bbae81462a22" containerName="revoke-keys" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978387 5515 cpu_manager.go:414] "containerMap: removing container" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978392 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978401 5515 cpu_manager.go:414] "containerMap: removing container" podUID="9870674e-3d75-4501-a5ce-4040294edd31" containerName="maas-api" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978406 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9870674e-3d75-4501-a5ce-4040294edd31" containerName="maas-api" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978415 5515 cpu_manager.go:414] "containerMap: removing container" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978420 5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978474 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a66893-8ec4-4286-b2de-52c5202295de" containerName="istio-proxy" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978482 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="11abbb0d-84ba-4a30-b0c3-c3ba1b5d7575" containerName="payload-pre-processing" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978488 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0abe68e-ab48-41a6-bcdb-ab612596352b" containerName="istio-proxy" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978495 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="75668c46-4685-4d6e-8a91-cc8f21364998" containerName="payload-processing" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978502 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b48c1f-cc31-4af2-8a47-bbae81462a22" containerName="revoke-keys" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978507 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9aa5bf2-ad4f-4bca-b301-8c9f1812b859" containerName="main" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978512 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9870674e-3d75-4501-a5ce-4040294edd31" containerName="maas-api" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978519 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a3c214-d603-45e7-8011-76b3e5d8bc67" containerName="payload-processing" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978525 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2516ca-0565-4a71-ba56-48e1b1aa3fc1" containerName="main" Oct 30 13:58:40.978599 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.978531 5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="48085915-c753-4106-9bc8-faa198f108f9" containerName="payload-processing" Oct 30 13:58:40.981486 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.981470 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z45hd/must-gather-2z97x" Oct 30 13:58:40.987442 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:40.987414 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:41.001994 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.001971 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z45hd\"/\"openshift-service-ca.crt\"" Oct 30 13:58:41.013944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.013918 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z45hd\"/\"kube-root-ca.crt\"" Oct 30 13:58:41.013944 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.013924 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z45hd\"/\"default-dockercfg-ln669\"" Oct 30 13:58:41.015533 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.015515 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z45hd/must-gather-2z97x"] Oct 30 13:58:41.068357 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.068331 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7hjjx/must-gather-6c8qw"] Oct 30 13:58:41.072214 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.072193 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fcw7g/must-gather-s5ccx"] Oct 30 13:58:41.072360 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.072340 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7hjjx/must-gather-6c8qw" Oct 30 13:58:41.085257 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.085235 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcw7g/must-gather-s5ccx" Oct 30 13:58:41.085378 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.085237 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/777e88ba-3005-47c9-9480-545a13789cca-must-gather-output\") pod \"must-gather-2z97x\" (UID: \"777e88ba-3005-47c9-9480-545a13789cca\") " pod="openshift-must-gather-z45hd/must-gather-2z97x" Oct 30 13:58:41.085544 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.085522 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4bb\" (UniqueName: \"kubernetes.io/projected/777e88ba-3005-47c9-9480-545a13789cca-kube-api-access-4g4bb\") pod \"must-gather-2z97x\" (UID: \"777e88ba-3005-47c9-9480-545a13789cca\") " pod="openshift-must-gather-z45hd/must-gather-2z97x" Oct 30 13:58:41.091369 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.091336 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7hjjx/must-gather-6c8qw"] Oct 30 13:58:41.094114 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.094094 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcw7g/must-gather-s5ccx"] Oct 30 13:58:41.094370 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.094329 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7hjjx\"/\"default-dockercfg-9z9dr\"" Oct 30 13:58:41.096890 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.096867 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7hjjx\"/\"openshift-service-ca.crt\"" Oct 30 13:58:41.097144 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.097123 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7hjjx\"/\"kube-root-ca.crt\"" Oct 30 13:58:41.101667 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.101644 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcw7g\"/\"kube-root-ca.crt\"" Oct 30 13:58:41.101775 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.101751 5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcw7g\"/\"openshift-service-ca.crt\"" Oct 30 13:58:41.186637 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.186608 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e-must-gather-output\") pod \"must-gather-6c8qw\" (UID: \"d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e\") " pod="openshift-must-gather-7hjjx/must-gather-6c8qw" Oct 30 13:58:41.186759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.186646 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/777e88ba-3005-47c9-9480-545a13789cca-must-gather-output\") pod \"must-gather-2z97x\" (UID: \"777e88ba-3005-47c9-9480-545a13789cca\") " pod="openshift-must-gather-z45hd/must-gather-2z97x" Oct 30 13:58:41.186759 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.186677 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkq2p\" (UniqueName: \"kubernetes.io/projected/d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e-kube-api-access-vkq2p\") pod \"must-gather-6c8qw\" (UID: \"d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e\") " pod="openshift-must-gather-7hjjx/must-gather-6c8qw" Oct 30 13:58:41.186844 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.186814 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f79s\" (UniqueName: \"kubernetes.io/projected/4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37-kube-api-access-8f79s\") pod \"must-gather-s5ccx\" (UID: \"4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37\") " pod="openshift-must-gather-fcw7g/must-gather-s5ccx" Oct 30 13:58:41.186880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.186848 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37-must-gather-output\") pod \"must-gather-s5ccx\" (UID: \"4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37\") " pod="openshift-must-gather-fcw7g/must-gather-s5ccx" Oct 30 13:58:41.186928 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.186916 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4bb\" (UniqueName: \"kubernetes.io/projected/777e88ba-3005-47c9-9480-545a13789cca-kube-api-access-4g4bb\") pod \"must-gather-2z97x\" (UID: \"777e88ba-3005-47c9-9480-545a13789cca\") " pod="openshift-must-gather-z45hd/must-gather-2z97x" Oct 30 13:58:41.186966 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.186931 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/777e88ba-3005-47c9-9480-545a13789cca-must-gather-output\") pod \"must-gather-2z97x\" (UID: \"777e88ba-3005-47c9-9480-545a13789cca\") " pod="openshift-must-gather-z45hd/must-gather-2z97x" Oct 30 13:58:41.192165 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.192149 5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fcw7g\"/\"default-dockercfg-znvlt\"" Oct 30 13:58:41.210011 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.209987 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4bb\" (UniqueName: \"kubernetes.io/projected/777e88ba-3005-47c9-9480-545a13789cca-kube-api-access-4g4bb\") pod \"must-gather-2z97x\" (UID: \"777e88ba-3005-47c9-9480-545a13789cca\") " pod="openshift-must-gather-z45hd/must-gather-2z97x" Oct 30 13:58:41.288392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.288328 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37-must-gather-output\") pod \"must-gather-s5ccx\" (UID: \"4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37\") " pod="openshift-must-gather-fcw7g/must-gather-s5ccx" Oct 30 13:58:41.288392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.288390 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e-must-gather-output\") pod \"must-gather-6c8qw\" (UID: \"d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e\") " pod="openshift-must-gather-7hjjx/must-gather-6c8qw" Oct 30 13:58:41.288574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.288424 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkq2p\" (UniqueName: \"kubernetes.io/projected/d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e-kube-api-access-vkq2p\") pod \"must-gather-6c8qw\" (UID: \"d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e\") " pod="openshift-must-gather-7hjjx/must-gather-6c8qw" Oct 30 13:58:41.288574 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.288467 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f79s\" (UniqueName: \"kubernetes.io/projected/4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37-kube-api-access-8f79s\") pod \"must-gather-s5ccx\" (UID: \"4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37\") " pod="openshift-must-gather-fcw7g/must-gather-s5ccx" Oct 30 13:58:41.288740 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.288718 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e-must-gather-output\") pod \"must-gather-6c8qw\" (UID: \"d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e\") " pod="openshift-must-gather-7hjjx/must-gather-6c8qw" Oct 30 13:58:41.288794 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.288718 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37-must-gather-output\") pod \"must-gather-s5ccx\" (UID: \"4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37\") " pod="openshift-must-gather-fcw7g/must-gather-s5ccx" Oct 30 13:58:41.296190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.296171 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z45hd/must-gather-2z97x" Oct 30 13:58:41.312084 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.312046 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkq2p\" (UniqueName: \"kubernetes.io/projected/d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e-kube-api-access-vkq2p\") pod \"must-gather-6c8qw\" (UID: \"d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e\") " pod="openshift-must-gather-7hjjx/must-gather-6c8qw" Oct 30 13:58:41.312368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.312350 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f79s\" (UniqueName: \"kubernetes.io/projected/4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37-kube-api-access-8f79s\") pod \"must-gather-s5ccx\" (UID: \"4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37\") " pod="openshift-must-gather-fcw7g/must-gather-s5ccx" Oct 30 13:58:41.381617 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.381587 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:41.389624 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.389602 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7hjjx/must-gather-6c8qw" Oct 30 13:58:41.402025 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.402008 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcw7g/must-gather-s5ccx" Oct 30 13:58:41.592470 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.592444 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z45hd/must-gather-2z97x"] Oct 30 13:58:41.593161 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:58:41.593141 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod777e88ba_3005_47c9_9480_545a13789cca.slice/crio-deba600eae5193d8cbcd4d7cffbae1583f5e25183c5a1414d129d9f2aa7e4b4d WatchSource:0}: Error finding container deba600eae5193d8cbcd4d7cffbae1583f5e25183c5a1414d129d9f2aa7e4b4d: Status 404 returned error can't find the container with id deba600eae5193d8cbcd4d7cffbae1583f5e25183c5a1414d129d9f2aa7e4b4d Oct 30 13:58:41.595069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.595053 5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:58:41.718882 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.718851 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7hjjx/must-gather-6c8qw"] Oct 30 13:58:41.723504 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:58:41.723471 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd389ebd5_debe_4dfc_a3b7_a2a2d3b2f36e.slice/crio-b26c9f1a02628efb6f10e242e1a5b840cf20cb37278f207fcad03d426f6f1b4c WatchSource:0}: Error finding container b26c9f1a02628efb6f10e242e1a5b840cf20cb37278f207fcad03d426f6f1b4c: Status 404 returned error can't find the container with id b26c9f1a02628efb6f10e242e1a5b840cf20cb37278f207fcad03d426f6f1b4c Oct 30 13:58:41.749981 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:41.749959 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcw7g/must-gather-s5ccx"] Oct 30 13:58:41.750506 ip-10-0-134-193 kubenswrapper[5515]: W1030 13:58:41.750485 5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e4e36bc_3dc7_4546_b4fc_fbaa7dba8c37.slice/crio-20bf18b1839234a412342f159d50ba1b3ed0bee85c7b213ab7904050fced085c WatchSource:0}: Error finding container 20bf18b1839234a412342f159d50ba1b3ed0bee85c7b213ab7904050fced085c: Status 404 returned error can't find the container with id 20bf18b1839234a412342f159d50ba1b3ed0bee85c7b213ab7904050fced085c Oct 30 13:58:42.362229 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:42.362143 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z45hd/must-gather-2z97x" event={"ID":"777e88ba-3005-47c9-9480-545a13789cca","Type":"ContainerStarted","Data":"deba600eae5193d8cbcd4d7cffbae1583f5e25183c5a1414d129d9f2aa7e4b4d"} Oct 30 13:58:42.363318 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:42.363256 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcw7g/must-gather-s5ccx" event={"ID":"4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37","Type":"ContainerStarted","Data":"20bf18b1839234a412342f159d50ba1b3ed0bee85c7b213ab7904050fced085c"} Oct 30 13:58:42.364324 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:42.364296 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7hjjx/must-gather-6c8qw" event={"ID":"d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e","Type":"ContainerStarted","Data":"b26c9f1a02628efb6f10e242e1a5b840cf20cb37278f207fcad03d426f6f1b4c"} Oct 30 13:58:43.379089 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:43.379049 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z45hd/must-gather-2z97x" event={"ID":"777e88ba-3005-47c9-9480-545a13789cca","Type":"ContainerStarted","Data":"2272ab87c04ba07cf0e432e0128bc8bf7d87c39938b03a0effe3d87818e773bf"} Oct 30 13:58:43.379539 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:43.379105 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z45hd/must-gather-2z97x" event={"ID":"777e88ba-3005-47c9-9480-545a13789cca","Type":"ContainerStarted","Data":"739731ba2d9bffc0773f33aa7e878239eef41350ea17ea481b95854ac833b678"} Oct 30 13:58:43.435758 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:43.435696 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z45hd/must-gather-2z97x" podStartSLOduration=2.651320044 podStartE2EDuration="3.435677732s" totalImagesPullingTime="784.357688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:43.435343332 +0000 UTC m=+5141.562144231" watchObservedRunningTime="2026-10-30 13:58:43.435677732 +0000 UTC m=+5141.562478631" Oct 30 13:58:46.589529 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:46.589454 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8fl8b_ecc1b4c7-ef74-4105-b613-2f53486e7aad/global-pull-secret-syncer/0.log" Oct 30 13:58:47.706039 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:47.706003 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" Oct 30 13:58:48.218225 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:48.209639 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-193.ec2.internal_f1f1b01344f2c1cc7bc384545bbc0754/haproxy/0.log" Oct 30 13:58:51.150917 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:51.150851 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:51.572726 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:51.572647 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:54.890902 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:54.890851 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-77jxw_3abefab9-6297-457f-b668-3282fd99560b/manager/0.log" Oct 30 13:58:55.951392 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:55.951347 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bd2wm_10a1b17f-494a-432f-910c-7c8ac33ad3ad/limitador/0.log" Oct 30 13:58:56.684299 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:56.684002 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcw7g/must-gather-s5ccx" event={"ID":"4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37","Type":"ContainerStarted","Data":"08e2fb5c546e4a9eca903f382614c94b3c3db4802f1911290ac4e08560f4b624"} Oct 30 13:58:56.687336 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:56.686445 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7hjjx/must-gather-6c8qw" event={"ID":"d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e","Type":"ContainerStarted","Data":"e7888527cd93d5059b33817a3c308b30480f49bd7a32fcbe635bdb33c263bac6"} Oct 30 13:58:57.714992 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.714952 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcw7g/must-gather-s5ccx" event={"ID":"4e4e36bc-3dc7-4546-b4fc-fbaa7dba8c37","Type":"ContainerStarted","Data":"60ca5211ae447ef622a76bd751eac6ff5d8ef3b69af7ce7b7f52bdb050a59805"} Oct 30 13:58:57.718523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.718494 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7hjjx/must-gather-6c8qw" event={"ID":"d389ebd5-debe-4dfc-a3b7-a2a2d3b2f36e","Type":"ContainerStarted","Data":"4aaf82711816a0911ddb9d5fabf29d33b4c54d249c455a27881679d0a9094c62"} Oct 30 13:58:57.758828 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.758765 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fcw7g/must-gather-s5ccx" podStartSLOduration=2.127124722 podStartE2EDuration="16.758746623s" totalImagesPullingTime="14.631621901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:57.757655939 +0000 UTC m=+5155.884456838" watchObservedRunningTime="2026-10-30 13:58:57.758746623 +0000 UTC m=+5155.885547517" Oct 30 13:58:57.784715 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.784668 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7dc73ed-382c-4a8e-a1e1-7621edf6cb79/alertmanager/0.log" Oct 30 13:58:57.798420 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.798350 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7hjjx/must-gather-6c8qw" podStartSLOduration=2.162740318 podStartE2EDuration="16.798329709s" totalImagesPullingTime="14.635589391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:57.797928241 +0000 UTC m=+5155.924729140" watchObservedRunningTime="2026-10-30 13:58:57.798329709 +0000 UTC m=+5155.925130608" Oct 30 13:58:57.895147 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.895103 5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k"] Oct 30 13:58:57.902061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.902032 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:57.918868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.918167 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7dc73ed-382c-4a8e-a1e1-7621edf6cb79/config-reloader/0.log" Oct 30 13:58:57.918868 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.918747 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k"] Oct 30 13:58:57.969340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.966657 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcxm\" (UniqueName: \"kubernetes.io/projected/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-kube-api-access-tpcxm\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:57.969340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.966693 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-lib-modules\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:57.969340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.966718 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-proc\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:57.969340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.966791 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-sys\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:57.969340 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:57.966912 5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-podres\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.051365 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.045973 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7dc73ed-382c-4a8e-a1e1-7621edf6cb79/kube-rbac-proxy-web/0.log" Oct 30 13:58:58.068289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.067764 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-podres\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.068289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.067844 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcxm\" (UniqueName: \"kubernetes.io/projected/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-kube-api-access-tpcxm\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.068289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.067872 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-lib-modules\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.068289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.067904 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-proc\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.068289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.067931 5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-sys\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.068289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.068049 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-sys\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.068289 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.068120 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-podres\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.068792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.068447 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-lib-modules\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.068792 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.068496 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-proc\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.101295 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.100849 5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcxm\" (UniqueName: \"kubernetes.io/projected/6f0aeff9-5e2b-42d0-a622-1d870ec3c029-kube-api-access-tpcxm\") pod \"perf-node-gather-daemonset-xlr6k\" (UID: \"6f0aeff9-5e2b-42d0-a622-1d870ec3c029\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.171595 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.171563 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7dc73ed-382c-4a8e-a1e1-7621edf6cb79/kube-rbac-proxy/0.log" Oct 30 13:58:58.228409 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.228334 5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:58.301892 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.300713 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7dc73ed-382c-4a8e-a1e1-7621edf6cb79/kube-rbac-proxy-metric/0.log" Oct 30 13:58:58.423208 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.423180 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7dc73ed-382c-4a8e-a1e1-7621edf6cb79/prom-label-proxy/0.log" Oct 30 13:58:58.555078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.550419 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7dc73ed-382c-4a8e-a1e1-7621edf6cb79/init-config-reloader/0.log" Oct 30 13:58:58.555078 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.552027 5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k"] Oct 30 13:58:58.734666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.734385 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" event={"ID":"6f0aeff9-5e2b-42d0-a622-1d870ec3c029","Type":"ContainerStarted","Data":"d683c524629a921be8e46981cf7343453e79433cff747f3e20f3e05dabf84107"} Oct 30 13:58:58.734666 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.734449 5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" event={"ID":"6f0aeff9-5e2b-42d0-a622-1d870ec3c029","Type":"ContainerStarted","Data":"69123bf40ff72e4771d85f1909193775775132609072eeae72e1ed29bead281a"} Oct 30 13:58:58.735215 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:58.734990 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:58:59.024248 ip-10-0-134-193 kubenswrapper[5515]: E1030 13:58:59.023952 5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8njh7" containerName="extractor" Oct 30 13:58:59.362131 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:59.362063 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8489fb66bd-4jwwm_a1dcad86-b55c-40ed-9280-a639b93d15d4/manager/0.log" Oct 30 13:58:59.393331 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:59.393302 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-q6hgs_475cbe1a-bcd8-4575-9c39-3c3394f062e4/monitoring-plugin/0.log" Oct 30 13:58:59.427362 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:59.427335 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8489fb66bd-4jwwm_a1dcad86-b55c-40ed-9280-a639b93d15d4/manager/0.log" Oct 30 13:58:59.588329 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:59.588248 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gr5gq_3e186430-fae4-49df-a08b-29ab2912ecd2/manager/0.log" Oct 30 13:58:59.620059 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:58:59.619971 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gr5gq_3e186430-fae4-49df-a08b-29ab2912ecd2/manager/0.log" Oct 30 13:59:00.326376 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:00.326349 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2vhb_5b73fd1c-6f57-4172-b98d-c8b8330efd83/node-exporter/0.log" Oct 30 13:59:00.547517 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:00.547485 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2vhb_5b73fd1c-6f57-4172-b98d-c8b8330efd83/kube-rbac-proxy/0.log" Oct 30 13:59:00.677415 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:00.677374 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2vhb_5b73fd1c-6f57-4172-b98d-c8b8330efd83/init-textfile/0.log" Oct 30 13:59:02.167625 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:02.167593 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-lvsdk_532d8fee-b407-48ed-b9f5-77b8ad7f7630/prometheus-operator/0.log" Oct 30 13:59:02.298325 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:02.298292 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-lvsdk_532d8fee-b407-48ed-b9f5-77b8ad7f7630/kube-rbac-proxy/0.log" Oct 30 13:59:02.562190 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:02.562111 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-874fd879-b99jw_98b4567f-f0d4-47a3-a8a3-3cac381e107d/telemeter-client/0.log" Oct 30 13:59:02.689448 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:02.689422 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-874fd879-b99jw_98b4567f-f0d4-47a3-a8a3-3cac381e107d/reload/0.log" Oct 30 13:59:02.827452 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:02.827373 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-874fd879-b99jw_98b4567f-f0d4-47a3-a8a3-3cac381e107d/kube-rbac-proxy/0.log" Oct 30 13:59:02.958606 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:02.958580 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b7b78d57-rlt5q_dbafdcd8-268f-4128-b17e-10d3d92e2227/thanos-query/0.log" Oct 30 13:59:03.094158 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:03.094084 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b7b78d57-rlt5q_dbafdcd8-268f-4128-b17e-10d3d92e2227/kube-rbac-proxy-web/0.log" Oct 30 13:59:03.303066 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:03.303035 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b7b78d57-rlt5q_dbafdcd8-268f-4128-b17e-10d3d92e2227/kube-rbac-proxy/0.log" Oct 30 13:59:03.470880 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:03.470853 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b7b78d57-rlt5q_dbafdcd8-268f-4128-b17e-10d3d92e2227/prom-label-proxy/0.log" Oct 30 13:59:03.598095 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:03.598065 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b7b78d57-rlt5q_dbafdcd8-268f-4128-b17e-10d3d92e2227/kube-rbac-proxy-rules/0.log" Oct 30 13:59:03.747136 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:03.747050 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b7b78d57-rlt5q_dbafdcd8-268f-4128-b17e-10d3d92e2227/kube-rbac-proxy-metrics/0.log" Oct 30 13:59:04.717222 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:04.717192 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8489fb66bd-4jwwm_a1dcad86-b55c-40ed-9280-a639b93d15d4/manager/0.log" Oct 30 13:59:04.755523 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:04.755497 5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" Oct 30 13:59:04.800367 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:04.800319 5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-xlr6k" podStartSLOduration=7.800305419 podStartE2EDuration="7.800305419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:58.781306776 +0000 UTC m=+5156.908107675" watchObservedRunningTime="2026-10-30 13:59:04.800305419 +0000 UTC m=+5162.927106317" Oct 30 13:59:04.871779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:04.871747 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8489fb66bd-4jwwm_a1dcad86-b55c-40ed-9280-a639b93d15d4/manager/0.log" Oct 30 13:59:04.911779 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:04.911730 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gr5gq_3e186430-fae4-49df-a08b-29ab2912ecd2/manager/0.log" Oct 30 13:59:05.047064 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:05.046975 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gr5gq_3e186430-fae4-49df-a08b-29ab2912ecd2/manager/0.log" Oct 30 13:59:08.231869 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:08.231823 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-fc75b545c-cdxqw_eb732958-cf4e-4753-b2e8-0382a471b48e/console/0.log" Oct 30 13:59:08.372460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:08.372428 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-s2gdv_983c8f5c-68cf-42a4-b931-4ceb2b7e1e77/download-server/0.log" Oct 30 13:59:09.933300 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:09.933246 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8489fb66bd-4jwwm_a1dcad86-b55c-40ed-9280-a639b93d15d4/manager/0.log" Oct 30 13:59:10.130657 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:10.130625 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gr5gq_3e186430-fae4-49df-a08b-29ab2912ecd2/manager/0.log" Oct 30 13:59:10.498995 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:10.498949 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8489fb66bd-4jwwm_a1dcad86-b55c-40ed-9280-a639b93d15d4/manager/0.log" Oct 30 13:59:10.696549 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:10.691803 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-gr5gq_3e186430-fae4-49df-a08b-29ab2912ecd2/manager/0.log" Oct 30 13:59:12.357809 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:12.357769 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dxl2t_dc939343-5581-480f-8c67-37ba0d1188c6/dns/0.log" Oct 30 13:59:12.474736 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:12.474665 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dxl2t_dc939343-5581-480f-8c67-37ba0d1188c6/kube-rbac-proxy/0.log" Oct 30 13:59:13.253203 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:13.253172 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2bkbt_24b12750-62c7-478f-bea4-8aee763363f3/dns-node-resolver/0.log" Oct 30 13:59:13.603069 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:13.602999 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59688578c6-wtg9m_b98339fa-aad1-4672-9cc8-5c09532a01e2/kube-auth-proxy/0.log" Oct 30 13:59:13.827003 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:13.826973 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-ljfbp_2876231a-a44b-4a8f-b462-4d30201f90d5/istio-proxy/0.log" Oct 30 13:59:13.895817 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:13.895687 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-q8crv_e8370493-9181-48ef-bd2e-b3eab3293f98/istio-proxy/0.log" Oct 30 13:59:14.139480 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:14.139450 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-ld25h_b4951dbc-6cb4-40ce-af70-a8d331e7c2b8/payload-pre-processing/0.log" Oct 30 13:59:14.213854 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:14.213774 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bvptt_7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca/payload-processing/0.log" Oct 30 13:59:15.238609 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:15.238575 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wsxsd_e218ebda-caba-45d2-963a-77b96b0af90f/node-ca/0.log" Oct 30 13:59:15.693741 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:15.693715 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59688578c6-wtg9m_b98339fa-aad1-4672-9cc8-5c09532a01e2/kube-auth-proxy/0.log" Oct 30 13:59:15.854108 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:15.847291 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-ljfbp_2876231a-a44b-4a8f-b462-4d30201f90d5/istio-proxy/0.log" Oct 30 13:59:15.922083 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:15.922040 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-q8crv_e8370493-9181-48ef-bd2e-b3eab3293f98/istio-proxy/0.log" Oct 30 13:59:16.167259 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:16.167227 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59688578c6-wtg9m_b98339fa-aad1-4672-9cc8-5c09532a01e2/kube-auth-proxy/0.log" Oct 30 13:59:16.212198 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:16.212170 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-ld25h_b4951dbc-6cb4-40ce-af70-a8d331e7c2b8/payload-pre-processing/0.log" Oct 30 13:59:16.288641 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:16.288608 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bvptt_7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca/payload-processing/0.log" Oct 30 13:59:16.321685 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:16.321648 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-ljfbp_2876231a-a44b-4a8f-b462-4d30201f90d5/istio-proxy/0.log" Oct 30 13:59:16.412368 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:16.412200 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-q8crv_e8370493-9181-48ef-bd2e-b3eab3293f98/istio-proxy/0.log" Oct 30 13:59:16.651061 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:16.650985 5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d44rw_5b35dcc8-6a2f-4cc9-a810-85ec056b5899/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:16.695405 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:16.695371 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-ld25h_b4951dbc-6cb4-40ce-af70-a8d331e7c2b8/payload-pre-processing/0.log" Oct 30 13:59:16.765460 ip-10-0-134-193 kubenswrapper[5515]: I1030 13:59:16.765423 5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bvptt_7d8cc90f-fd5b-4b71-ab60-5c2af7ee35ca/payload-processing/0.log"